Jul 16 17:34:33.118783 ip-10-0-142-99 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 17:34:34.071689 ip-10-0-142-99 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:34.071689 ip-10-0-142-99 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 17:34:34.071689 ip-10-0-142-99 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:34.071689 ip-10-0-142-99 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 17:34:34.071689 ip-10-0-142-99 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:34.073707 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.073618    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 17:34:34.078616 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078600    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.078616 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078617    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078623    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078628    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078631    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078634    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078638    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078640    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078643    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078646    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078649    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078652    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078655    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078658    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078661    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078666    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078670    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078672    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078675    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078682    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078685    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.078686 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078688    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078692    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078695    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078698    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078701    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078704    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078706    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078709    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078712    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078715    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078717    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078720    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078723    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078726    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078728    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078731    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078734    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078736    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078739    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078742    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.079172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078745    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078747    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078750    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078753    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078756    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078758    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078761    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078764    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078766    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078769    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078772    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078775    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078778    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078780    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078784    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078787    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078790    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078793    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078795    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078798    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.079688 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078801    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078803    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078806    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078809    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078812    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078814    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078817    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078819    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078822    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078825    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078835    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078838    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078841    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078844    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078847    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078851    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078854    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078856    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078859    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.080191 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078862    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078866    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078869    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078872    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078875    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078878    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078882    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078886    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078889    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078892    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078895    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078898    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078901    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078904    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078906    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078909    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078911    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078914    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.078917    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.080691 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079695    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079703    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079707    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079710    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079714    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079718    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079722    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079725    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079728    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079732    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079736    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079740    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079743    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079746    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079749    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079752    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079756    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079759    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079762    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.081170 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079765    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079768    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079770    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079773    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079776    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079779    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079781    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079784    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079787    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079789    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079792    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079794    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079797    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079800    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079802    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079805    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079808    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079811    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079813    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079820    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.081668 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079824    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079827    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079830    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079833    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079836    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079839    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079841    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079844    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079847    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079850    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079853    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079855    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079858    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079861    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079863    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079866    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079870    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079874    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079877    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079880    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.082169 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079883    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079886    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079888    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079891    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079894    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079896    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079899    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079902    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079905    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079907    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079911    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079914    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079917    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079922    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079925    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079928    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079931    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079935    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079938    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.082677 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079942    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079945    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079948    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079951    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079954    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079957    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079960    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079963    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079966    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079969    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079972    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079975    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079978    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079980    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079983    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079986    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079988    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079991    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079994    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079996    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.083172 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.079999    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080380    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080385    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080388    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080391    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080395    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080398    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080401    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080404    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080407    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080410    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080413    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080415    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080418    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080423    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080427    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080430    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080433    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080436    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.083687 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080439    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080443    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080445    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080448    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080451    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080454    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080457    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080459    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080462    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080465    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080467    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080470    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080473    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080476    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080479    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080481    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080484    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080487    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080491    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.084206 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080494    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080499    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080503    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080506    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080509    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080512    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080515    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080518    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080521    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080524    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080527    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080530    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080532    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080535    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080538    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080541    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080543    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080546    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080548    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080551    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.084702 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080554    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080556    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080559    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080562    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080567    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080570    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080572    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080590    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080598    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080601    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080604    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080607    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080610    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080613    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080616    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080618    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080621    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080624    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080627    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.085204 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080630    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080635    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080639    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080643    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080646    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080650    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080653    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080656    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080659    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080662    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080665    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080668    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080672    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080675    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080678    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080681    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080684    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080687    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080690    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.085836 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080693    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080696    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.080701    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080775    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080784    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080795    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080802    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080808    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080813    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080820    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080828    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080833    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080838    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080841    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080845    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080848    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080852    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080855    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080858    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080861    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080865    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080869    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080872    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080876    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 17:34:34.086313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080879    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080883    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080887    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080890    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080894    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080897    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080900    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080903    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080907    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080910    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080914    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080919    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080923    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080926    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080929    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080932    2577 flags.go:64] FLAG: --enable-server="true"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080937    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080941    2577 flags.go:64] FLAG: --event-burst="100"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080945    2577 flags.go:64] FLAG: --event-qps="50"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080948    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080956    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080959    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080966    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080969    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080972    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 17:34:34.086928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080976    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080979    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080982    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080986    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080989    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080992    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080995    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.080998    2577 flags.go:64] FLAG: --feature-gates=""
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081002    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081005    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081008    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081011    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081015    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081018    2577 flags.go:64] FLAG: --help="false"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081021    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081024    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081027    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081032    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081036    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081039    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081042    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081045    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081048    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 17:34:34.087594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081052    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081056    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081059    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081062    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081067    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081070    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081073    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081076    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081079    2577 flags.go:64] FLAG: --lock-file=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081082    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081085    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081088    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081094    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081097    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081100    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081103    2577 flags.go:64] FLAG: --logging-format="text"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081106    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081110    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081113    2577 flags.go:64] FLAG: --manifest-url=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081116    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081120    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081124    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081128    2577 flags.go:64] FLAG: --max-pods="110"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081131    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081134    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 17:34:34.088205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081137    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081142    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081145    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081148    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081151    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081158    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081161    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081165    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081168    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081172    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081178    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081182    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081185    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081188    2577 flags.go:64] FLAG: --port="10250"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081191    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081194    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03449fc2d2e0ed957"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081198    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081201    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081204    2577 flags.go:64] FLAG: --register-node="true"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081207    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081211    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081214    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081217    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081220    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081224    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 17:34:34.088862 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081227    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081230    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081233    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081236    2577 flags.go:64] FLAG: --runonce="false"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081239    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081242    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081245    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081248    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081251    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081255    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081258    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081261    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081264    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081267    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081270    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081273    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081276    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081280    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081283    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081289    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081292    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081295    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081299    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081302    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081306    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 17:34:34.089523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081309    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081312    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081315    2577 flags.go:64] FLAG: --v="2"
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081319    2577 flags.go:64] FLAG: --version="false"
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081323    2577 flags.go:64] FLAG: --vmodule=""
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081327    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.081331    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081450    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081454    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081457    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081460    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081463    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081467    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081470    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081473    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081476    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081479    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081482    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081485    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081488    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081490    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.090174 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081493    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081496    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081500    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081502    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081506    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081509    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081512    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081514    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081518    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081522    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081527    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081531    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081534    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081537    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081540    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081543    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081546    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081549    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081552    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.090774 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081555    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081558    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081563    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081566    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081569    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081572    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081590    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081593    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081596    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081599    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081601    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081604    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081607    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081610    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081613    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081615    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081618    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081622    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081625    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.091260 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081628    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081630    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081633    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081636    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081638    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081641    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081644    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081647    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081650    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081652    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081655    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081657    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081660    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081663    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081665    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081668    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081671    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081676    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081679    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081682    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.091779 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081685    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081688    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081691    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081694    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081696    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081699    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081702    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081705    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081708    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081712    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081715    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081719    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081722    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081725    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081728    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081730    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081733    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081736    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081738    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081741    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.092271 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081744    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081747    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081750    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081752    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081755    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.081758    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.082439    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.089806    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.089822    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.090790    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.090798    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090859    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090865    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090869    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.092811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090872    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090875    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090878    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090881    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090884    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090887    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090890    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090892    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090897    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090901    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090904    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090908    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090912    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090915    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090917    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090920    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090923    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090926    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090929    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090931    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.093177 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090935    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090937    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090940    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090943    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090946    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090950    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090954    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090957    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090960    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090964    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090969    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090972    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090975    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090978    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090981    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090984    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090987    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090990    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090993    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090995    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.093692 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.090998    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091001    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091004    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091007    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091010    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091013    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091015    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091018    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091021    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091024    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091027    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091029    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091033    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091035    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091038    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091041    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091044    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091047    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091051    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091054    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.094207 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091057    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091060    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091062    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091065    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091069    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091072    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091075    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091078    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091081    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091085    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091088    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091091    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091094    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091097    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091100    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091104    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091107    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091110    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091113    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091116    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.094793 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091119    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091122    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091125    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091128    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091132    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091134    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091137    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091140    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091144    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091147    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091151    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091154    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091158    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091161    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091164    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.095323 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091167    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.091172    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091289    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091294    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091297    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091300    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091304    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091306    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091310    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091313    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091316    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091319    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091322    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091324    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091327    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:34.095733 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091330    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091333    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091335    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091338    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091341    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091344    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091346    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091349    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091351    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091354    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091357    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091362    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091366    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091369    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091372    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091376    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091379    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091382    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091385    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:34.096121 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091388    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091391    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091394    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091397    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091400    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091403    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091405    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091408    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091411    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091413    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091416    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091419    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091421    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091424    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091427    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091430    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091432    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091436    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091438    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091441    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:34.096617 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091444    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091446    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091449    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091452    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091456    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091460    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091463    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091466    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091469    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091475    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091478    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091481    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091484    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091487    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091490    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091492    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091495    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091498    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091501    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:34.097128 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091503    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091506    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091509    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091511    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091514    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091517    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091519    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091522    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091525    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091527    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091531    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091534    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091536    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091539    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091541    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091544    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091547    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091549    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091552    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091555    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:34.097623 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091557    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091564    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091569    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091572    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091587    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091593    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:34.091597    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.091604    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.091709    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.093889    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.095599    2577 server.go:1019] "Starting client certificate rotation"
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.095702    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:34.098151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.095739    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:34.098808 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.098794    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:34.102281 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.102257    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:34.114633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.114608    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 16 17:34:34.116782 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.116767    2577 log.go:25] "Validated CRI v1 image API"
Jul 16 17:34:34.117679 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.117663    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 17:34:34.120196 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.120176    2577 fs.go:135] Filesystem UUIDs: map[117c03ba-18f1-4673-85a7-b1502de1c7c0:/dev/nvme0n1p3 33a94c94-7c85-4504-a750-0f0ad8d4e479:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 16 17:34:34.120274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.120194    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 17:34:34.124678 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.124556    2577 manager.go:216] Machine: {Timestamp:2026-07-16 17:34:34.124059155 +0000 UTC m=+0.880467210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096011 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec235c16421ddffb2ef67df97d138ecf SystemUUID:ec235c16-421d-dffb-2ef6-7df97d138ecf BootID:2ba9db02-2470-4678-85ca-c8bd6d7a86b1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:74:90:ba:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:74:90:ba:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:ca:43:ba:a6:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 17:34:34.124678 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.124672    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 17:34:34.124835 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.124775    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 17:34:34.129076 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.129051    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 17:34:34.129210 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.129079    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 17:34:34.130307 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.130292    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:34.130480 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.130469    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 17:34:34.130516 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.130483    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 17:34:34.130516 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.130491    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:34.131380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.131369    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:34.133106 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.133095    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 17:34:34.136482 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136470    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:34.136641 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136632    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 17:34:34.136680 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136663    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 17:34:34.136720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136683    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 17:34:34.136720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136699    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 17:34:34.136720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136708    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 16 17:34:34.136720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.136717    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 17:34:34.137067 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.137055    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:34.137099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.137076    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:34.139303 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.139282    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 17:34:34.141394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141378    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 17:34:34.141466 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141409    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 17:34:34.141574 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141563    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 17:34:34.141636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141598    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 17:34:34.141636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141606    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 17:34:34.141636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141612    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 17:34:34.141636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141621    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 17:34:34.141636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.141626    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 17:34:34.142338 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142328    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 17:34:34.142338 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142338    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 17:34:34.142408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142346    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 17:34:34.142408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142352    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 17:34:34.142408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142361    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 17:34:34.142408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.142369    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 17:34:34.144012 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.143979    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 17:34:34.144012 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.143989    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 17:34:34.148505 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.148487    2577 server.go:1268] "Started kubelet"
Jul 16 17:34:34.148627 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.148578    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 17:34:34.148689 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.148649    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 17:34:34.148739 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.148604    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 17:34:34.149383 ip-10-0-142-99 systemd[1]: Started Kubernetes Kubelet.
Jul 16 17:34:34.149977 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.149952    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 17:34:34.150279 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.150262    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 17:34:34.151423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.151403    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 17:34:34.152482 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.152452    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 17:34:34.152581 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.152454    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 17:34:34.152667 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.152643    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 17:34:34.156550 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.156530    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 17:34:34.156550 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.156543    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:34.156712 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.156668    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.157352 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.157334    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 17:34:34.157451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.157335    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 17:34:34.157451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.157370    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 17:34:34.157451 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.155879    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c2d59d579a59cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-16 17:34:34.148452812 +0000 UTC m=+0.904860867,LastTimestamp:2026-07-16 17:34:34.148452812 +0000 UTC m=+0.904860867,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}"
Jul 16 17:34:34.157451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.157429    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 17:34:34.157451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.157440    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 17:34:34.158098 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158073    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 17:34:34.158098 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158094    2577 factory.go:55] Registering systemd factory
Jul 16 17:34:34.158230 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158104    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 16 17:34:34.158323 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158309    2577 factory.go:153] Registering CRI-O factory
Jul 16 17:34:34.158323 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158323    2577 factory.go:223] Registration of the crio container factory successfully
Jul 16 17:34:34.158410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158342    2577 factory.go:103] Registering Raw factory
Jul 16 17:34:34.158410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158353    2577 manager.go:1196] Started watching for new ooms in manager
Jul 16 17:34:34.158752 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.158735    2577 manager.go:349] Starting recovery of all containers
Jul 16 17:34:34.165335 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.165299    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 17:34:34.165335 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.165305    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 17:34:34.170148 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.170122    2577 manager.go:354] Recovery completed
Jul 16 17:34:34.176101 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.176087    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.178147 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178122    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.178208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178161    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.178208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178172    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.178756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178739    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 17:34:34.178756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178756    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 17:34:34.178869 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178774    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.178869 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.178776    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:34.180050 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180027    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.180110 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180060    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.180110 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180070    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.180401 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180389    2577 policy_none.go:49] "None policy: Start"
Jul 16 17:34:34.180432 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180406    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 17:34:34.180432 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.180417    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 17:34:34.180960 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.180899    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c2d59d595f7262  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-16 17:34:34.178146914 +0000 UTC m=+0.934554971,LastTimestamp:2026-07-16 17:34:34.178146914 +0000 UTC m=+0.934554971,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}"
Jul 16 17:34:34.181634 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.181623    2577 policy_none.go:47] "Start"
Jul 16 17:34:34.190851 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.190790    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c2d59d595fbe01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-16 17:34:34.178166273 +0000 UTC m=+0.934574330,LastTimestamp:2026-07-16 17:34:34.178166273 +0000 UTC m=+0.934574330,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}"
Jul 16 17:34:34.197571 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.197550    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkgr6"
Jul 16 17:34:34.204853 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.204638    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-99.ec2.internal.18c2d59d595fe3e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-99.ec2.internal,UID:ip-10-0-142-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-99.ec2.internal,},FirstTimestamp:2026-07-16 17:34:34.178175969 +0000 UTC m=+0.934584024,LastTimestamp:2026-07-16 17:34:34.178175969 +0000 UTC m=+0.934584024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-99.ec2.internal,}"
Jul 16 17:34:34.206084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.206068    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkgr6"
Jul 16 17:34:34.223154 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223140    2577 manager.go:337] "Starting Device Plugin manager"
Jul 16 17:34:34.223223 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.223169    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 17:34:34.223223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223180    2577 server.go:87] "Starting device plugin registration server"
Jul 16 17:34:34.223428 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223415    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 17:34:34.223469 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223430    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 17:34:34.223557 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223540    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 17:34:34.223640 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223627    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 17:34:34.223702 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.223643    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 17:34:34.224147 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.224130    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 17:34:34.224217 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.224184    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.297441 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.297401    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 17:34:34.298750 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.298638    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 17:34:34.298750 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.298683    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 17:34:34.298750 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.298710    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 17:34:34.298909 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.298750    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 17:34:34.302116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.302099    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:34.323817 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.323792    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.324669 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.324651    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.324742 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.324687    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.324742 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.324704    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.324742 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.324731    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.336038 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.336020    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.336086 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.336040    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-99.ec2.internal\": node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.364557 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.362592    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.399558 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.399462    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"]
Jul 16 17:34:34.399558 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.399534    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.401513 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.401496    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.401616 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.401540    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.401616 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.401551    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.402965 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.402953    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.403087 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.403180 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403096    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.403678 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403659    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.403766 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403690    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.403766 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403706    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.403766 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403727    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.403766 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403744    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.403766 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.403754    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.405142 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.405129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.405203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.405154    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:34.412846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.412832    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:34.412919 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.412859    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:34.412919 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.412874    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:34.426146 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.426121    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-99.ec2.internal\" not found" node="ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.432185 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.432170    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-99.ec2.internal\" not found" node="ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.459367 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.459348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c50785a27e73d42664b01bc31a18dac-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8c50785a27e73d42664b01bc31a18dac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.459471 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.459376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.459471 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.459395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.463681 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.463665    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.560300 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.560300 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.560484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c50785a27e73d42664b01bc31a18dac-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8c50785a27e73d42664b01bc31a18dac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.560484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c50785a27e73d42664b01bc31a18dac-config\") pod \"kube-apiserver-proxy-ip-10-0-142-99.ec2.internal\" (UID: \"8c50785a27e73d42664b01bc31a18dac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.560484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.560484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.560397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ca3c2571621c1d1b304562d5377290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal\" (UID: \"d3ca3c2571621c1d1b304562d5377290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.564711 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.564697    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.665668 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.665610    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.729541 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.729513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.736016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:34.736001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:34.766230 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.766203    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.866364 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.866321    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:34.967252 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:34.967190    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:35.068349 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.068305    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-99.ec2.internal\" not found"
Jul 16 17:34:35.073053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.073030    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:35.095421 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.095396    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 17:34:35.095573 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.095552    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 17:34:35.095616 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.095576    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 17:34:35.140042 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.140007    2577 apiserver.go:52] "Watching apiserver"
Jul 16 17:34:35.157270 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.157245    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:35.157270 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.157255    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:35.157821 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.157801    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 17:34:35.158941 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.158921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-785gt","openshift-cluster-node-tuning-operator/tuned-874qv","openshift-dns/node-resolver-nfsf7","openshift-multus/multus-additional-cni-plugins-5bskg","openshift-multus/multus-zbvvt","openshift-multus/network-metrics-daemon-tgz85","openshift-network-diagnostics/network-check-target-9tvfn","kube-system/konnectivity-agent-4hmls","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w","openshift-image-registry/node-ca-fq5kb","openshift-network-operator/iptables-alerter-shrmd","openshift-ovn-kubernetes/ovnkube-node-vhthf"]
Jul 16 17:34:35.160523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.160495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.160638 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.160617    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:35.162787 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.162760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.165180 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.165160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.165492 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.165472    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.165561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.165546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kv2zl\""
Jul 16 17:34:35.165626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.165613    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.166554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.166539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.167466 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.167444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.167545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.167468    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xm2dz\""
Jul 16 17:34:35.167545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.167487    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.167901 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.167886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.168563 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.168530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjg4m\""
Jul 16 17:34:35.168893 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.168878    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 17:34:35.169039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169016    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 17:34:35.169143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 17:34:35.169143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.169143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169098    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.169307 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.169360 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.169317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:35.169868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.169832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 17:34:35.170107 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.170093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jks2b\""
Jul 16 17:34:35.170519 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.170503    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:35.170622 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.170524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:35.170685 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.170621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:35.171198 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.171183    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:35.171274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.171258    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"
Jul 16 17:34:35.171894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.171874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.173548 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.173535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.174209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.174192    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 17:34:35.174352 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.174327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l55hw\""
Jul 16 17:34:35.174706 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.174690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 17:34:35.174974 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.174957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.175646 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.175626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 17:34:35.175980 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.175940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.176116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.176015    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-98kn6\""
Jul 16 17:34:35.176555 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.176147    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.177008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.176991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.177173 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.177150    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 17:34:35.177445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.177429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.177525 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.177514    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rzn8\""
Jul 16 17:34:35.177612 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.177553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.178572 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.178556    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal"]
Jul 16 17:34:35.178660 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.178649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.179326 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.179310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jqjqk\""
Jul 16 17:34:35.179406 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.179334    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.179634 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.179619    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.179744 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.179730    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 17:34:35.179865 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.179849    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:35.180016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.180002    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal"]
Jul 16 17:34:35.180832 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.180818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 17:34:35.180915 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.180901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 17:34:35.181014 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.180998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 17:34:35.181080 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.181002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nczq4\""
Jul 16 17:34:35.181080 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.181002    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 17:34:35.181221 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.181207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 17:34:35.181433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.181421    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 17:34:35.196592 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.196566    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fs9w6"
Jul 16 17:34:35.203904 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.203888    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fs9w6"
Jul 16 17:34:35.209685 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.209658    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:29:34 +0000 UTC" deadline="2028-04-11 17:56:43.103077486 +0000 UTC"
Jul 16 17:34:35.209685 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.209685    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15240h22m7.893395372s"
Jul 16 17:34:35.258343 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.258324    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 17:34:35.263488 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.263570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqqt\" (UniqueName: \"kubernetes.io/projected/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-kube-api-access-6nqqt\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.263570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.263570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.263570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8f2\" (UniqueName: \"kubernetes.io/projected/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-kube-api-access-kx8f2\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-etc-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-hostroot\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-device-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29e7a4a8-bacb-4180-adf9-908d44d2d172-iptables-alerter-script\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.263723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-systemd-units\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-modprobe-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-sys\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-systemd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-cni-binary-copy\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-multus\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-multus-certs\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658633e6-f491-4234-b88e-482a65cdab95-tmp-dir\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.263894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e7a4a8-bacb-4180-adf9-908d44d2d172-host-slash\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-netd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-registration-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.263994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-os-release\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn86c\" (UniqueName: \"kubernetes.io/projected/ae688bb9-84d2-4be9-add5-340b787a7b4e-kube-api-access-xn86c\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-node-log\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-kubernetes\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-system-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-netns\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-conf-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cbd90d-23de-4ce3-b04c-f18a1bee0051-agent-certs\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-bin\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-system-cni-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cbd90d-23de-4ce3-b04c-f18a1bee0051-konnectivity-ca\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-dbus\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysconfig\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-k8s-cni-cncf-io\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fcm\" (UniqueName: \"kubernetes.io/projected/29e7a4a8-bacb-4180-adf9-908d44d2d172-kube-api-access-m4fcm\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.264400 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovn-node-metrics-cert\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-var-lib-kubelet\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-socket-dir-parent\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-systemd\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-kubelet\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-socket-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-log-socket\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-run\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-os-release\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-daemon-config\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-etc-kubernetes\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ncl\" (UniqueName: \"kubernetes.io/projected/658633e6-f491-4234-b88e-482a65cdab95-kube-api-access-z5ncl\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-env-overrides\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-script-lib\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-kubelet-config\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-etc-tuned\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.264826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/658633e6-f491-4234-b88e-482a65cdab95-hosts-file\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6fs\" (UniqueName: \"kubernetes.io/projected/8290a32d-9c64-4537-bf74-7794bc98094b-kube-api-access-9z6fs\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-serviceca\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-netns\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-bin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cnibin\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-kubelet\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-ovn\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29c9r\" (UniqueName: \"kubernetes.io/projected/9e650ce2-5cf1-4786-8905-9c4775e5c64c-kube-api-access-29c9r\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-slash\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.264999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-conf\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-host\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.265274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-host\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-var-lib-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-config\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-lib-modules\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-tmp\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-cnibin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-sys-fs\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtsf\" (UniqueName: \"kubernetes.io/projected/510501f8-c06c-456b-af8c-4ae7d057bb87-kube-api-access-pwtsf\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.265801 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.265228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvct4\" (UniqueName: \"kubernetes.io/projected/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-kube-api-access-xvct4\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.365583 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-cnibin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-sys-fs\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtsf\" (UniqueName: \"kubernetes.io/projected/510501f8-c06c-456b-af8c-4ae7d057bb87-kube-api-access-pwtsf\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvct4\" (UniqueName: \"kubernetes.io/projected/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-kube-api-access-xvct4\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-cnibin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.365754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-sys-fs\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqqt\" (UniqueName: \"kubernetes.io/projected/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-kube-api-access-6nqqt\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8f2\" (UniqueName: \"kubernetes.io/projected/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-kube-api-access-kx8f2\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-etc-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.365972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-hostroot\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-device-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.366039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29e7a4a8-bacb-4180-adf9-908d44d2d172-iptables-alerter-script\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-systemd-units\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-hostroot\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-modprobe-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-device-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-sys\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-systemd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-systemd-units\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-cni-binary-copy\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-modprobe-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-sys\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-multus\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-systemd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-multus\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-multus-certs\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658633e6-f491-4234-b88e-482a65cdab95-tmp-dir\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e7a4a8-bacb-4180-adf9-908d44d2d172-host-slash\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.366607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-multus-certs\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-netd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-registration-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-netd\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-os-release\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-registration-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn86c\" (UniqueName: \"kubernetes.io/projected/ae688bb9-84d2-4be9-add5-340b787a7b4e-kube-api-access-xn86c\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-node-log\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-kubernetes\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-etc-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-system-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-netns\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658633e6-f491-4234-b88e-482a65cdab95-tmp-dir\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-conf-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.367481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-os-release\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cbd90d-23de-4ce3-b04c-f18a1bee0051-agent-certs\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29e7a4a8-bacb-4180-adf9-908d44d2d172-host-slash\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29e7a4a8-bacb-4180-adf9-908d44d2d172-iptables-alerter-script\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-bin\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-system-cni-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-netns\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-system-cni-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-cni-binary-copy\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-node-log\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-kubernetes\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cbd90d-23de-4ce3-b04c-f18a1bee0051-konnectivity-ca\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-system-cni-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-dbus\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysconfig\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368349 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-cni-bin\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367000    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-k8s-cni-cncf-io\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysconfig\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-dbus\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-run-k8s-cni-cncf-io\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.366903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-conf-dir\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fcm\" (UniqueName: \"kubernetes.io/projected/29e7a4a8-bacb-4180-adf9-908d44d2d172-kube-api-access-m4fcm\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovn-node-metrics-cert\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-d\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-var-lib-kubelet\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-socket-dir-parent\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-var-lib-kubelet\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-systemd\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.368955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-kubelet\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-socket-dir-parent\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-socket-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-kubelet\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-log-socket\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-systemd\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-run\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-os-release\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-log-socket\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38cbd90d-23de-4ce3-b04c-f18a1bee0051-konnectivity-ca\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-run\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-os-release\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-socket-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-daemon-config\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-etc-kubernetes\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ncl\" (UniqueName: \"kubernetes.io/projected/658633e6-f491-4234-b88e-482a65cdab95-kube-api-access-z5ncl\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-etc-kubernetes\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-env-overrides\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.369796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-script-lib\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-kubelet-config\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-etc-tuned\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e6da779-aad7-4536-a3aa-30ae688b10a4-kubelet-config\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/658633e6-f491-4234-b88e-482a65cdab95-hosts-file\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6fs\" (UniqueName: \"kubernetes.io/projected/8290a32d-9c64-4537-bf74-7794bc98094b-kube-api-access-9z6fs\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-serviceca\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-netns\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae688bb9-84d2-4be9-add5-340b787a7b4e-multus-daemon-config\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-bin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cnibin\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.367993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-kubelet\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-ovn\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/658633e6-f491-4234-b88e-482a65cdab95-hosts-file\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.370831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29c9r\" (UniqueName: \"kubernetes.io/projected/9e650ce2-5cf1-4786-8905-9c4775e5c64c-kube-api-access-29c9r\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-slash\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-env-overrides\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-run-ovn\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-slash\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-conf\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/510501f8-c06c-456b-af8c-4ae7d057bb87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-script-lib\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-etc-sysctl-conf\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae688bb9-84d2-4be9-add5-340b787a7b4e-host-var-lib-cni-bin\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-run-netns\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-cnibin\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-host-kubelet\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.368388    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.368445    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.868429879 +0000 UTC m=+2.624837941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.371546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-host\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-serviceca\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-host\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-host\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-var-lib-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-config\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-var-lib-openvswitch\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-host\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.368992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.369020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-lib-modules\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.369045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-tmp\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.369149    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.369169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8290a32d-9c64-4537-bf74-7794bc98094b-lib-modules\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.369214    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.869197407 +0000 UTC m=+2.625605450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.369408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovnkube-config\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.369759    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.371012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-etc-tuned\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.372179 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.371179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-ovn-node-metrics-cert\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.373044 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.371347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38cbd90d-23de-4ce3-b04c-f18a1bee0051-agent-certs\") pod \"konnectivity-agent-4hmls\" (UID: \"38cbd90d-23de-4ce3-b04c-f18a1bee0051\") " pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.373044 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.371441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8290a32d-9c64-4537-bf74-7794bc98094b-tmp\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.376489 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.376464    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:35.376606 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.376493    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:35.376606 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.376508    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:35.376606 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.376573    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.87655759 +0000 UTC m=+2.632965654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:35.378865 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.378809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqqt\" (UniqueName: \"kubernetes.io/projected/e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac-kube-api-access-6nqqt\") pod \"multus-additional-cni-plugins-5bskg\" (UID: \"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac\") " pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.379337 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.379293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvct4\" (UniqueName: \"kubernetes.io/projected/3b8f3865-fae0-4b1b-8a8f-92f8776798b6-kube-api-access-xvct4\") pod \"node-ca-fq5kb\" (UID: \"3b8f3865-fae0-4b1b-8a8f-92f8776798b6\") " pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.379542 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.379521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fcm\" (UniqueName: \"kubernetes.io/projected/29e7a4a8-bacb-4180-adf9-908d44d2d172-kube-api-access-m4fcm\") pod \"iptables-alerter-shrmd\" (UID: \"29e7a4a8-bacb-4180-adf9-908d44d2d172\") " pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.379814 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.379784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8f2\" (UniqueName: \"kubernetes.io/projected/ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd-kube-api-access-kx8f2\") pod \"ovnkube-node-vhthf\" (UID: \"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.380248 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.380228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ncl\" (UniqueName: \"kubernetes.io/projected/658633e6-f491-4234-b88e-482a65cdab95-kube-api-access-z5ncl\") pod \"node-resolver-nfsf7\" (UID: \"658633e6-f491-4234-b88e-482a65cdab95\") " pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.380321 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.380289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtsf\" (UniqueName: \"kubernetes.io/projected/510501f8-c06c-456b-af8c-4ae7d057bb87-kube-api-access-pwtsf\") pod \"aws-ebs-csi-driver-node-f7g8w\" (UID: \"510501f8-c06c-456b-af8c-4ae7d057bb87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.380718 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.380697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn86c\" (UniqueName: \"kubernetes.io/projected/ae688bb9-84d2-4be9-add5-340b787a7b4e-kube-api-access-xn86c\") pod \"multus-zbvvt\" (UID: \"ae688bb9-84d2-4be9-add5-340b787a7b4e\") " pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.380825 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.380806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6fs\" (UniqueName: \"kubernetes.io/projected/8290a32d-9c64-4537-bf74-7794bc98094b-kube-api-access-9z6fs\") pod \"tuned-874qv\" (UID: \"8290a32d-9c64-4537-bf74-7794bc98094b\") " pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.382022 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.382003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29c9r\" (UniqueName: \"kubernetes.io/projected/9e650ce2-5cf1-4786-8905-9c4775e5c64c-kube-api-access-29c9r\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.489472 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.489414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-874qv"
Jul 16 17:34:35.503621 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.503601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nfsf7"
Jul 16 17:34:35.510509 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.510486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bskg"
Jul 16 17:34:35.525460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.525439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zbvvt"
Jul 16 17:34:35.533487 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.533470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:35.542314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.542296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w"
Jul 16 17:34:35.553312 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.553290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fq5kb"
Jul 16 17:34:35.557053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.557036    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:35.572990 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.572968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shrmd"
Jul 16 17:34:35.578092 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.578075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:34:35.858190 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:35.858147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c50785a27e73d42664b01bc31a18dac.slice/crio-7799a9adce239e4775ebc2312a50d92f2e3236962433ae9797233271488f9ffd WatchSource:0}: Error finding container 7799a9adce239e4775ebc2312a50d92f2e3236962433ae9797233271488f9ffd: Status 404 returned error can't find the container with id 7799a9adce239e4775ebc2312a50d92f2e3236962433ae9797233271488f9ffd
Jul 16 17:34:35.869008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.868281    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:34:35.872292 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.872268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:35.872403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.872371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:35.872492 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.872470    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.872492 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.872487    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.872614 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.872543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:36.872520521 +0000 UTC m=+3.628928563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.872614 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.872559    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:34:36.872551646 +0000 UTC m=+3.628959687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.876676 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:35.876654    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e7a4a8_bacb_4180_adf9_908d44d2d172.slice/crio-31713f98b80507e0ad24667e0c47ee304c7b863e6ee42993237ef906dab4935f WatchSource:0}: Error finding container 31713f98b80507e0ad24667e0c47ee304c7b863e6ee42993237ef906dab4935f: Status 404 returned error can't find the container with id 31713f98b80507e0ad24667e0c47ee304c7b863e6ee42993237ef906dab4935f
Jul 16 17:34:35.902539 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:35.902515    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cbbeb8_9ff3_4e97_bc8d_5a7c784415ac.slice/crio-63c4bcf45763a7f532a208972a206d5ef0875d4b3a38003191345c6511aac864 WatchSource:0}: Error finding container 63c4bcf45763a7f532a208972a206d5ef0875d4b3a38003191345c6511aac864: Status 404 returned error can't find the container with id 63c4bcf45763a7f532a208972a206d5ef0875d4b3a38003191345c6511aac864
Jul 16 17:34:35.972736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.972707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:35.972845 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.972833    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:35.972907 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.972853    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:35.972907 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.972864    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:35.973019 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:35.972919    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:36.972901228 +0000 UTC m=+3.729309273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:35.973019 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:35.972938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658633e6_f491_4234_b88e_482a65cdab95.slice/crio-9f4d9643824313cb1cb4e0f6877c66761975717b144be33d3a839f2a15e3c01e WatchSource:0}: Error finding container 9f4d9643824313cb1cb4e0f6877c66761975717b144be33d3a839f2a15e3c01e: Status 404 returned error can't find the container with id 9f4d9643824313cb1cb4e0f6877c66761975717b144be33d3a839f2a15e3c01e
Jul 16 17:34:35.973284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:35.973268    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:35.973362 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:35.973329    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510501f8_c06c_456b_af8c_4ae7d057bb87.slice/crio-23fceed0313473070046ba73d9ed4ede073a974212c4956420bbcfd72c172b50 WatchSource:0}: Error finding container 23fceed0313473070046ba73d9ed4ede073a974212c4956420bbcfd72c172b50: Status 404 returned error can't find the container with id 23fceed0313473070046ba73d9ed4ede073a974212c4956420bbcfd72c172b50
Jul 16 17:34:36.040479 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:36.040455    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae688bb9_84d2_4be9_add5_340b787a7b4e.slice/crio-1d9ce841e3ef0fb81e94f67478ea065da0ddfbfd03e13530cf135df96ae4896d WatchSource:0}: Error finding container 1d9ce841e3ef0fb81e94f67478ea065da0ddfbfd03e13530cf135df96ae4896d: Status 404 returned error can't find the container with id 1d9ce841e3ef0fb81e94f67478ea065da0ddfbfd03e13530cf135df96ae4896d
Jul 16 17:34:36.207597 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.207489    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:35 +0000 UTC" deadline="2028-03-28 04:16:59.611236932 +0000 UTC"
Jul 16 17:34:36.207597 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.207552    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14890h42m23.40368857s"
Jul 16 17:34:36.299713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.299683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:36.299863 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.299835    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:36.303863 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.303810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w" event={"ID":"510501f8-c06c-456b-af8c-4ae7d057bb87","Type":"ContainerStarted","Data":"23fceed0313473070046ba73d9ed4ede073a974212c4956420bbcfd72c172b50"}
Jul 16 17:34:36.306931 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.306906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nfsf7" event={"ID":"658633e6-f491-4234-b88e-482a65cdab95","Type":"ContainerStarted","Data":"9f4d9643824313cb1cb4e0f6877c66761975717b144be33d3a839f2a15e3c01e"}
Jul 16 17:34:36.308576 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.308530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerStarted","Data":"63c4bcf45763a7f532a208972a206d5ef0875d4b3a38003191345c6511aac864"}
Jul 16 17:34:36.310019 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.309987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shrmd" event={"ID":"29e7a4a8-bacb-4180-adf9-908d44d2d172","Type":"ContainerStarted","Data":"31713f98b80507e0ad24667e0c47ee304c7b863e6ee42993237ef906dab4935f"}
Jul 16 17:34:36.311207 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.311184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" event={"ID":"8c50785a27e73d42664b01bc31a18dac","Type":"ContainerStarted","Data":"7799a9adce239e4775ebc2312a50d92f2e3236962433ae9797233271488f9ffd"}
Jul 16 17:34:36.312373 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.312353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zbvvt" event={"ID":"ae688bb9-84d2-4be9-add5-340b787a7b4e","Type":"ContainerStarted","Data":"1d9ce841e3ef0fb81e94f67478ea065da0ddfbfd03e13530cf135df96ae4896d"}
Jul 16 17:34:36.500467 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:36.500434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf6e4cf_7c73_44b1_bc4c_da6c15cdfedd.slice/crio-91a5d9854a366e10eeb43119485a321612df423ff7cf9d95a635204279cb355d WatchSource:0}: Error finding container 91a5d9854a366e10eeb43119485a321612df423ff7cf9d95a635204279cb355d: Status 404 returned error can't find the container with id 91a5d9854a366e10eeb43119485a321612df423ff7cf9d95a635204279cb355d
Jul 16 17:34:36.879891 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.879786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:36.879891 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.879846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:36.880113 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.880005    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:36.880113 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.880005    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:36.880228 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.880080    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:38.880060033 +0000 UTC m=+5.636468078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:36.880228 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.880147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:34:38.880132901 +0000 UTC m=+5.636540945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:36.978484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.978453    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:36.980942 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:36.980901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:36.981093 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.981066    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:36.981093 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.981086    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:36.981204 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.981100    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:36.981204 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:36.981202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:38.981182328 +0000 UTC m=+5.737590372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:37.208749 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:37.208607    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:35 +0000 UTC" deadline="2028-01-28 09:59:48.00807638 +0000 UTC"
Jul 16 17:34:37.208749 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:37.208663    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h25m10.799416905s"
Jul 16 17:34:37.299526 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:37.299481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:37.299731 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:37.299659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:37.300125 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:37.300099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:37.300241 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:37.300212    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:37.317707 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:37.317574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"91a5d9854a366e10eeb43119485a321612df423ff7cf9d95a635204279cb355d"}
Jul 16 17:34:38.300059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:38.299903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:38.300059 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:38.300037    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:38.889493 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:38.889463    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cbd90d_23de_4ce3_b04c_f18a1bee0051.slice/crio-a999d84279f7371f95bb99abf9cf7ad0941e0d7bd10f6eccfb547966e2538458 WatchSource:0}: Error finding container a999d84279f7371f95bb99abf9cf7ad0941e0d7bd10f6eccfb547966e2538458: Status 404 returned error can't find the container with id a999d84279f7371f95bb99abf9cf7ad0941e0d7bd10f6eccfb547966e2538458
Jul 16 17:34:38.900081 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:38.899790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:38.900081 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:38.899880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:38.900081 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:38.899996    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:38.900081 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:38.900023    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:38.900081 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:38.900075    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:42.900053445 +0000 UTC m=+9.656461501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:38.900476 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:38.900095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:34:42.900084383 +0000 UTC m=+9.656492429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:38.945075 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:38.945037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8f3865_fae0_4b1b_8a8f_92f8776798b6.slice/crio-a979294d156b8b934141cc38900c9ca68e8cb700455773567b4613cbce9a34fb WatchSource:0}: Error finding container a979294d156b8b934141cc38900c9ca68e8cb700455773567b4613cbce9a34fb: Status 404 returned error can't find the container with id a979294d156b8b934141cc38900c9ca68e8cb700455773567b4613cbce9a34fb
Jul 16 17:34:38.955530 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:38.955316    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ca3c2571621c1d1b304562d5377290.slice/crio-eccd472b9fe6553dd6ece36c8a74c38a7781aab042a23f8a65a323f6939d146c WatchSource:0}: Error finding container eccd472b9fe6553dd6ece36c8a74c38a7781aab042a23f8a65a323f6939d146c: Status 404 returned error can't find the container with id eccd472b9fe6553dd6ece36c8a74c38a7781aab042a23f8a65a323f6939d146c
Jul 16 17:34:39.000252 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.000226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:39.000400 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.000375    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:39.000457 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.000407    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:39.000457 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.000422    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:39.000544 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.000482    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:43.00046016 +0000 UTC m=+9.756868228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:39.299363 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.298929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:39.299363 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.299050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:39.299363 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.299114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:39.299363 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:39.299234    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:39.323545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.323497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"d3ca3c2571621c1d1b304562d5377290","Type":"ContainerStarted","Data":"eccd472b9fe6553dd6ece36c8a74c38a7781aab042a23f8a65a323f6939d146c"}
Jul 16 17:34:39.325625 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.325599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hmls" event={"ID":"38cbd90d-23de-4ce3-b04c-f18a1bee0051","Type":"ContainerStarted","Data":"a999d84279f7371f95bb99abf9cf7ad0941e0d7bd10f6eccfb547966e2538458"}
Jul 16 17:34:39.329953 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.329920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" event={"ID":"8c50785a27e73d42664b01bc31a18dac","Type":"ContainerStarted","Data":"cccf484bff1d7fc966895828f448a63f52b53a9f53fc51e637128bf213df98bb"}
Jul 16 17:34:39.331999 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:39.331969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fq5kb" event={"ID":"3b8f3865-fae0-4b1b-8a8f-92f8776798b6","Type":"ContainerStarted","Data":"a979294d156b8b934141cc38900c9ca68e8cb700455773567b4613cbce9a34fb"}
Jul 16 17:34:40.299266 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:40.299231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:40.299454 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:40.299368    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:40.337294 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:40.337257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nfsf7" event={"ID":"658633e6-f491-4234-b88e-482a65cdab95","Type":"ContainerStarted","Data":"1845cde06577f837fd01d997ce72ccdb70f5fe884bf8d80e3efeb986251d5b3b"}
Jul 16 17:34:40.359453 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:40.359399    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-99.ec2.internal" podStartSLOduration=5.359377519 podStartE2EDuration="5.359377519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:39.354613705 +0000 UTC m=+6.111021772" watchObservedRunningTime="2026-07-16 17:34:40.359377519 +0000 UTC m=+7.115785580"
Jul 16 17:34:41.299518 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:41.299482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:41.299709 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:41.299639    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:41.299873 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:41.299482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:41.300008 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:41.299921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:41.341658 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:41.341594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shrmd" event={"ID":"29e7a4a8-bacb-4180-adf9-908d44d2d172","Type":"ContainerStarted","Data":"e9c1403050b8f6aa3cc2437429a27f30f9b570753480018087657aec937fe466"}
Jul 16 17:34:41.366489 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:41.366445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nfsf7" podStartSLOduration=3.456417594 podStartE2EDuration="6.366428039s" totalImagesPullingTime="2.910010445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:40.360758507 +0000 UTC m=+7.117166574" watchObservedRunningTime="2026-07-16 17:34:41.366428039 +0000 UTC m=+8.122836098"
Jul 16 17:34:41.366977 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:41.366943    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-shrmd" podStartSLOduration=3.364523265 podStartE2EDuration="6.366935376s" totalImagesPullingTime="3.002412111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:41.366917471 +0000 UTC m=+8.123325537" watchObservedRunningTime="2026-07-16 17:34:41.366935376 +0000 UTC m=+8.123343441"
Jul 16 17:34:42.300032 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:42.299941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:42.300225 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:42.300091    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:42.934253 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:42.934207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:42.934253 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:42.934255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:42.934854 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:42.934384    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:42.934854 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:42.934454    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:50.934430288 +0000 UTC m=+17.690838336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:42.934854 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:42.934526    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:42.934854 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:42.934585    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:34:50.934570893 +0000 UTC m=+17.690978941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:43.034971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:43.034938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:43.035132 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.035106    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:43.035194 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.035151    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:43.035194 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.035166    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:43.035293 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.035220    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:51.035203512 +0000 UTC m=+17.791611555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:43.154621 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:34:43.151459    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8290a32d_9c64_4537_bf74_7794bc98094b.slice/crio-509d332e868b95fea53c3d787609557f9a35b49ec9025fafcd049540e5abbd26 WatchSource:0}: Error finding container 509d332e868b95fea53c3d787609557f9a35b49ec9025fafcd049540e5abbd26: Status 404 returned error can't find the container with id 509d332e868b95fea53c3d787609557f9a35b49ec9025fafcd049540e5abbd26
Jul 16 17:34:43.299966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:43.299885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:43.300117 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.300031    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:43.300117 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:43.300102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:43.300239 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:43.300164    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:43.348471 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:43.348437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-874qv" event={"ID":"8290a32d-9c64-4537-bf74-7794bc98094b","Type":"ContainerStarted","Data":"509d332e868b95fea53c3d787609557f9a35b49ec9025fafcd049540e5abbd26"}
Jul 16 17:34:44.300514 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:44.300041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:44.300514 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:44.300141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:45.299402 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:45.299304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:45.299574 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:45.299439    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:45.299574 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:45.299484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:45.299671 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:45.299587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:46.299134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:46.299080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:46.299587 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:46.299216    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:47.299274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:47.299240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:47.299274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:47.299266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:47.299677 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:47.299359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:47.299677 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:47.299485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:48.299323 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:48.299286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:48.299802 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:48.299424    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:49.299128 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:49.299100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:49.299269 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:49.299106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:49.299269 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:49.299207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:49.299351 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:49.299276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:50.303095 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:50.303067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:50.303530 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:50.303236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:50.998542 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:50.998496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:50.998542 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:50.998544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:50.998775 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:50.998660    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:50.998775 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:50.998716    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:06.99869682 +0000 UTC m=+33.755104859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:50.998775 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:50.998660    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:50.998909 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:50.998813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:06.99878996 +0000 UTC m=+33.755198010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:51.099358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:51.099325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:51.099540 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.099493    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:51.099540 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.099533    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:51.099627 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.099547    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:51.099627 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.099607    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:07.099590707 +0000 UTC m=+33.855998763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:51.298955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:51.298872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:51.298955 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:51.298915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:51.299176 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.299002    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:51.299231 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:51.299173    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:52.301884 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:52.301854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:52.302427 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:52.301972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:53.299748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:53.299716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:53.299926 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:53.299714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:53.299926 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:53.299845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:53.300011 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:53.299935    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:54.300306 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:54.300272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:54.300766 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:54.300359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:55.299086 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:55.299051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:55.299313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:55.299089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:55.299313 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:55.299177    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:55.299415 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:55.299319    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:56.299434 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.299239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:56.300284 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:56.299542    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:56.375296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.375052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w" event={"ID":"510501f8-c06c-456b-af8c-4ae7d057bb87","Type":"ContainerStarted","Data":"847f09a028b8b486e2a124ade5a7325e71e473d49756e80356f81613c0b097f5"}
Jul 16 17:34:56.376642 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.376617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"9f9a4e5ff54a235a5109c00dabf090964a2521175be1287c04cd77ae2d672c64"}
Jul 16 17:34:56.378286 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.378230    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3ca3c2571621c1d1b304562d5377290" containerID="03d0096f8dc0e9872eed2993d788139c6528d88eb570b0679bb044abcc98d605" exitCode=0
Jul 16 17:34:56.378371 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.378299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"d3ca3c2571621c1d1b304562d5377290","Type":"ContainerDied","Data":"03d0096f8dc0e9872eed2993d788139c6528d88eb570b0679bb044abcc98d605"}
Jul 16 17:34:56.379913 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.379894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hmls" event={"ID":"38cbd90d-23de-4ce3-b04c-f18a1bee0051","Type":"ContainerStarted","Data":"8eb18796485e55f9995640cb96f40a42cdc7385f4840c1bde7bf81efa73c53ff"}
Jul 16 17:34:56.381532 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.381431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-874qv" event={"ID":"8290a32d-9c64-4537-bf74-7794bc98094b","Type":"ContainerStarted","Data":"1d8e7e63de8023a132f9780ac63ed9ae947f8ada5e23203ba13b594933978392"}
Jul 16 17:34:56.383319 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.383296    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="fc2ef465730ca01da3834279ec9e18828dfc10add45e61fb6664c22b86e31690" exitCode=0
Jul 16 17:34:56.383408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.383378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"fc2ef465730ca01da3834279ec9e18828dfc10add45e61fb6664c22b86e31690"}
Jul 16 17:34:56.385812 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.385539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fq5kb" event={"ID":"3b8f3865-fae0-4b1b-8a8f-92f8776798b6","Type":"ContainerStarted","Data":"5bf58df40839cce526b4a1d9d79b32448e2c43c3661d02b44fef8cab9f8a1ccd"}
Jul 16 17:34:56.387310 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.387290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zbvvt" event={"ID":"ae688bb9-84d2-4be9-add5-340b787a7b4e","Type":"ContainerStarted","Data":"1db099e715dca40283a3ee73cbe547d48df5102af77927323f3b2a1bef7e4ab7"}
Jul 16 17:34:56.445060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.445006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hmls" podStartSLOduration=4.299396628 podStartE2EDuration="21.444986837s" totalImagesPullingTime="17.145590209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:56.409569953 +0000 UTC m=+23.165978015" watchObservedRunningTime="2026-07-16 17:34:56.444986837 +0000 UTC m=+23.201394902"
Jul 16 17:34:56.475800 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.475764    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zbvvt" podStartSLOduration=1.478890242 podStartE2EDuration="21.47574801s" totalImagesPullingTime="19.996857768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:56.444937764 +0000 UTC m=+23.201345826" watchObservedRunningTime="2026-07-16 17:34:56.47574801 +0000 UTC m=+23.232156069"
Jul 16 17:34:56.476110 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.476089    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fq5kb" podStartSLOduration=4.786419984 podStartE2EDuration="21.476081163s" totalImagesPullingTime="16.689661179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:56.475965692 +0000 UTC m=+23.232373753" watchObservedRunningTime="2026-07-16 17:34:56.476081163 +0000 UTC m=+23.232489224"
Jul 16 17:34:56.554081 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:56.554043    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-874qv" podStartSLOduration=8.65573809 podStartE2EDuration="21.554030517s" totalImagesPullingTime="12.898292427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:56.512795811 +0000 UTC m=+23.269203873" watchObservedRunningTime="2026-07-16 17:34:56.554030517 +0000 UTC m=+23.310438579"
Jul 16 17:34:57.298971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.298937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:57.298971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.298970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:57.299188 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:57.299070    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:57.299252 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:57.299198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:57.392578 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.392488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"5b3acbd0ad3acbacd0c2fd6abd215a9d699dfe27adc8def8035bf07cc5850152"}
Jul 16 17:34:57.393030 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.392585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"ca3237a9cbfc7eb650f420e36d79e140a84312aaf236ba813af9171db6f8f024"}
Jul 16 17:34:57.393030 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.392634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"a63868bd27ce6083c76ff2ad9da5a340ce0f4df7d92f25d75440eea9004d8627"}
Jul 16 17:34:57.393030 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.392679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"81d93d2ff6159ce4b2823aae1308a697353801f8cec60f37a2d8bd6d221801f0"}
Jul 16 17:34:57.393030 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:57.392725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"9380f9460948090344e8e5ac0a974c1000642b4933b868335e9135f459d195f9"}
Jul 16 17:34:58.299979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.299743    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:34:58.300143 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:58.300088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:34:58.396951 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.396919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" event={"ID":"d3ca3c2571621c1d1b304562d5377290","Type":"ContainerStarted","Data":"dde39f7b9c5c0b00f3bd79eee7c73e5b96a447abc9741d58a1ae89c862c1cd56"}
Jul 16 17:34:58.480441 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.480411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:58.480987 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.480962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:58.511352 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.511301    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-99.ec2.internal" podStartSLOduration=23.511281018 podStartE2EDuration="23.511281018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:58.423131916 +0000 UTC m=+25.179539978" watchObservedRunningTime="2026-07-16 17:34:58.511281018 +0000 UTC m=+25.267689082"
Jul 16 17:34:58.721427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:58.721403    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 17:34:59.238140 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.238048    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:34:58.721418975Z","UUID":"163111c8-aa71-483d-be3c-11f9b029ccef","Handler":null,"Name":"","Endpoint":""}
Jul 16 17:34:59.239462 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.239442    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 17:34:59.239614 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.239483    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 17:34:59.299021 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.298998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:34:59.299141 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.299039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:34:59.299196 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:59.299130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:34:59.299259 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:34:59.299218    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:34:59.400788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.400754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w" event={"ID":"510501f8-c06c-456b-af8c-4ae7d057bb87","Type":"ContainerStarted","Data":"fd3bf9c0619e9b9deec095d40082d0ee7a60e8c99be8b79f4b07a2d360c82867"}
Jul 16 17:34:59.401314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.400842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:34:59.401388 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:34:59.401371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hmls"
Jul 16 17:35:00.299164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:00.299136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:00.299277 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:00.299255    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:35:00.405975 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:00.405943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"d65ebf419d82413bc836335c1aa763663770980f33321402dff0df72ef370fbe"}
Jul 16 17:35:01.299540 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:01.299516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:01.299630 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:01.299518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:01.299692 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:01.299633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:35:01.299748 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:01.299689    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:35:01.409831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:01.409808    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="42a90e379aa44000d548ed7f057927a6b4531c253b430291159e34766f2839c6" exitCode=0
Jul 16 17:35:01.410220 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:01.409878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"42a90e379aa44000d548ed7f057927a6b4531c253b430291159e34766f2839c6"}
Jul 16 17:35:01.412105 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:01.412086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w" event={"ID":"510501f8-c06c-456b-af8c-4ae7d057bb87","Type":"ContainerStarted","Data":"7fcd27de55df5f8339cb4615ee74e36889ce541a5e1fec41ccc4f7d7fe1f340b"}
Jul 16 17:35:02.299012 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:02.298986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:02.299166 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:02.299090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:35:03.299645 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.299407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:03.300087 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.299410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:03.300087 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:03.299734    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:35:03.300087 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:03.299870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:35:03.417632 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.417601    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="c276a465a06408e78bea392906a0ee00e611638d90fadafe4cc336e6152ac4c0" exitCode=0
Jul 16 17:35:03.417764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.417688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"c276a465a06408e78bea392906a0ee00e611638d90fadafe4cc336e6152ac4c0"}
Jul 16 17:35:03.421384 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.421361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" event={"ID":"ddf6e4cf-7c73-44b1-bc4c-da6c15cdfedd","Type":"ContainerStarted","Data":"0cbc626b58e1a129f9008ae9c2d4dc346a29552d6b1b8d7bb44cea3a83045392"}
Jul 16 17:35:03.421646 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.421627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:03.421741 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.421694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:03.435976 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.435958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:03.447354 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:03.447285    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7g8w" podStartSLOduration=16.462567069 podStartE2EDuration="28.447273681s" totalImagesPullingTime="11.984706612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:01.494982092 +0000 UTC m=+28.251390155" watchObservedRunningTime="2026-07-16 17:35:03.447273681 +0000 UTC m=+30.203681812"
Jul 16 17:35:04.302531 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:04.300529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:04.302531 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:04.300767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:35:04.423892 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:04.423867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:04.438015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:04.437991    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:04.504369 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:04.504302    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf" podStartSLOduration=9.944417028 podStartE2EDuration="29.504281812s" totalImagesPullingTime="19.559864784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:03.489191794 +0000 UTC m=+30.245599855" watchObservedRunningTime="2026-07-16 17:35:04.504281812 +0000 UTC m=+31.260689873"
Jul 16 17:35:05.036369 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.036335    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9tvfn"]
Jul 16 17:35:05.036558 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.036455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:05.036633 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:05.036558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:35:05.039097 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.039072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgz85"]
Jul 16 17:35:05.039220 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.039191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:05.039321 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:05.039299    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:35:05.041630 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.041608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-785gt"]
Jul 16 17:35:05.041731 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.041697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:05.041784 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:05.041767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:35:05.428383 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.428310    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="ee89c9c96138ffd450983fecdd43b2c2d2440ab5d71c921e2fb73f9ee336e2e8" exitCode=0
Jul 16 17:35:05.428839 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:05.428382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"ee89c9c96138ffd450983fecdd43b2c2d2440ab5d71c921e2fb73f9ee336e2e8"}
Jul 16 17:35:06.299961 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:06.299798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:06.300112 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:06.300041    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:35:07.022621 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:07.022577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:07.023048 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:07.022632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:07.023048 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.022703    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:07.023048 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.022724    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:07.023048 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.022778    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:35:39.022757514 +0000 UTC m=+65.779165559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:07.023048 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.022800    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret podName:4e6da779-aad7-4536-a3aa-30ae688b10a4 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:39.022789908 +0000 UTC m=+65.779197953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret") pod "global-pull-secret-syncer-785gt" (UID: "4e6da779-aad7-4536-a3aa-30ae688b10a4") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:07.123702 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:07.123597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:07.123865 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.123796    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:35:07.123865 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.123830    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:35:07.123865 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.123845    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jwhbw for pod openshift-network-diagnostics/network-check-target-9tvfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:07.123996 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.123909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw podName:5c6175b9-2bb5-4715-a352-d0a7db815981 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:39.123890207 +0000 UTC m=+65.880298251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwhbw" (UniqueName: "kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw") pod "network-check-target-9tvfn" (UID: "5c6175b9-2bb5-4715-a352-d0a7db815981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:07.299278 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:07.299243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:07.299523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:07.299244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:07.299523 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.299351    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785gt" podUID="4e6da779-aad7-4536-a3aa-30ae688b10a4"
Jul 16 17:35:07.299523 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:07.299444    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9tvfn" podUID="5c6175b9-2bb5-4715-a352-d0a7db815981"
Jul 16 17:35:08.299311 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:08.299286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:08.299771 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:08.299390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgz85" podUID="9e650ce2-5cf1-4786-8905-9c4775e5c64c"
Jul 16 17:35:09.031319 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.031292    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-99.ec2.internal" event="NodeReady"
Jul 16 17:35:09.031512 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.031425    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 17:35:09.072324 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.072299    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cb59898df-5dfdd"]
Jul 16 17:35:09.105648 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.105627    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cb59898df-5dfdd"]
Jul 16 17:35:09.105788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.105659    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j6rxh"]
Jul 16 17:35:09.105788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.105762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.110875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.109963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 17:35:09.110875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.110249    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 17:35:09.110875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.110424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wsnnh\""
Jul 16 17:35:09.110875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.110643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 17:35:09.119949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.119926    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 17:35:09.121673 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.121653    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dmmld"]
Jul 16 17:35:09.121792 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.121770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.125324 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.125309    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 17:35:09.125427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.125373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2d8t\""
Jul 16 17:35:09.125795 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.125776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 17:35:09.141736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.141713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.141736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.141711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6rxh"]
Jul 16 17:35:09.141879 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.141804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dmmld"]
Jul 16 17:35:09.146909 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.146861    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 17:35:09.146909 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.146874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 17:35:09.147051 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.146933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-27svs\""
Jul 16 17:35:09.149474 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.149457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 17:35:09.237748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjn6\" (UniqueName: \"kubernetes.io/projected/0a3dce21-336b-42fb-81b0-59104d77c969-kube-api-access-zgjn6\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a3dce21-336b-42fb-81b0-59104d77c969-cert\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5885fde-e5b3-423c-9442-990136b5a521-tmp-dir\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pbv\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.237888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.237990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.238019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5885fde-e5b3-423c-9442-990136b5a521-config-volume\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.238093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.238143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.238134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mfm\" (UniqueName: \"kubernetes.io/projected/c5885fde-e5b3-423c-9442-990136b5a521-kube-api-access-l4mfm\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.238354 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.238191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.299690 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.299634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:09.299690 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.299667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:09.303386 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.303365    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:09.303386 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.303378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28hgh\""
Jul 16 17:35:09.303575 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.303365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:09.303748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.303729    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:09.339035 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a3dce21-336b-42fb-81b0-59104d77c969-cert\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.339148 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5885fde-e5b3-423c-9442-990136b5a521-tmp-dir\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.339148 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pbv\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339259 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339259 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.339358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5885fde-e5b3-423c-9442-990136b5a521-config-volume\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mfm\" (UniqueName: \"kubernetes.io/projected/c5885fde-e5b3-423c-9442-990136b5a521-kube-api-access-l4mfm\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjn6\" (UniqueName: \"kubernetes.io/projected/0a3dce21-336b-42fb-81b0-59104d77c969-kube-api-access-zgjn6\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339582 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.339876 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.339698    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:09.339876 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.339760    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:09.8397369 +0000 UTC m=+36.596144952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:09.339876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5885fde-e5b3-423c-9442-990136b5a521-tmp-dir\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.339876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.339866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.340178 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.340156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.340267 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.340243    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:09.340267 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.340255    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:09.340375 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.340302    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:09.840286941 +0000 UTC m=+36.596694981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:09.340375 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.340333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5885fde-e5b3-423c-9442-990136b5a521-config-volume\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.340482 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.340429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.343684 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.343663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a3dce21-336b-42fb-81b0-59104d77c969-cert\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.343771 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.343666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.343771 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.343712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.354209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.354164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.354209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.354172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjn6\" (UniqueName: \"kubernetes.io/projected/0a3dce21-336b-42fb-81b0-59104d77c969-kube-api-access-zgjn6\") pod \"ingress-canary-dmmld\" (UID: \"0a3dce21-336b-42fb-81b0-59104d77c969\") " pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.354209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.354200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pbv\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.355525 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.355490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mfm\" (UniqueName: \"kubernetes.io/projected/c5885fde-e5b3-423c-9442-990136b5a521-kube-api-access-l4mfm\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.450894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.450859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dmmld"
Jul 16 17:35:09.621608 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.621493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:09.842677 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.842642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:09.842870 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.842708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:09.842870 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.842819    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:09.842962 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.842879    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:09.842962 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.842893    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:10.842870796 +0000 UTC m=+37.599278836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:09.842962 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.842899    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:09.842962 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:09.842963    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:10.842946428 +0000 UTC m=+37.599354486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:09.967794 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:09.967765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:10.108388 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.108182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dmmld"]
Jul 16 17:35:10.177997 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:35:10.177961    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3dce21_336b_42fb_81b0_59104d77c969.slice/crio-5633c5977d30f9119c6088bbdc093bd61d9614c2cc149b8d963970d4bf5bcffd WatchSource:0}: Error finding container 5633c5977d30f9119c6088bbdc093bd61d9614c2cc149b8d963970d4bf5bcffd: Status 404 returned error can't find the container with id 5633c5977d30f9119c6088bbdc093bd61d9614c2cc149b8d963970d4bf5bcffd
Jul 16 17:35:10.299583 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.299561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:10.303380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.303362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:10.303755 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.303523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4hnr\""
Jul 16 17:35:10.442067 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.442041    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="7c76ee9e3a1032ae48b489eb1735f6e28d27c65d84721f0693192c2f3223e96d" exitCode=0
Jul 16 17:35:10.442170 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.442113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"7c76ee9e3a1032ae48b489eb1735f6e28d27c65d84721f0693192c2f3223e96d"}
Jul 16 17:35:10.443255 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.443222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dmmld" event={"ID":"0a3dce21-336b-42fb-81b0-59104d77c969","Type":"ContainerStarted","Data":"5633c5977d30f9119c6088bbdc093bd61d9614c2cc149b8d963970d4bf5bcffd"}
Jul 16 17:35:10.852445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.852414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:10.852615 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:10.852478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:10.852683 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:10.852612    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:10.852683 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:10.852665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:12.852652448 +0000 UTC m=+39.609060489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:10.852789 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:10.852615    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:10.852789 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:10.852714    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:10.852878 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:10.852788    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:12.852768954 +0000 UTC m=+39.609177008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:11.449293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:11.449254    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac" containerID="7dacdb7d4d69d3e9aa66a53697f644162009833b828607bd048ed287338da470" exitCode=0
Jul 16 17:35:11.449710 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:11.449308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerDied","Data":"7dacdb7d4d69d3e9aa66a53697f644162009833b828607bd048ed287338da470"}
Jul 16 17:35:11.551144 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:11.551121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:12.175379 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.175348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:12.455788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.455719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bskg" event={"ID":"e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac","Type":"ContainerStarted","Data":"6e9fe106ef9408b9406aa52233944c987d36f79d71f11da05ba1eb12832d98fe"}
Jul 16 17:35:12.491950 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.491904    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5bskg" podStartSLOduration=5.413062172 podStartE2EDuration="37.491889216s" totalImagesPullingTime="32.078827044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:12.491544565 +0000 UTC m=+39.247952628" watchObservedRunningTime="2026-07-16 17:35:12.491889216 +0000 UTC m=+39.248297285"
Jul 16 17:35:12.628426 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.628397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:12.700160 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.699963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:12.870230 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.870197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:12.870377 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.870242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:12.870377 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:12.870355    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:12.870461 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:12.870419    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.870404269 +0000 UTC m=+43.626812313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:12.870461 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:12.870358    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:12.870558 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:12.870463    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:12.870558 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:12.870524    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.870498213 +0000 UTC m=+43.626906269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:12.989390 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:12.989350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:13.459791 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:13.459759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dmmld" event={"ID":"0a3dce21-336b-42fb-81b0-59104d77c969","Type":"ContainerStarted","Data":"c0ee077f596bbf0e0fdfde88e00ece98ed9cfb98545836119c66b5c62a6bab1c"}
Jul 16 17:35:13.481480 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:13.481440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dmmld" podStartSLOduration=2.104399865 podStartE2EDuration="4.481425338s" totalImagesPullingTime="2.377025473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:13.479252009 +0000 UTC m=+40.235660072" watchObservedRunningTime="2026-07-16 17:35:13.481425338 +0000 UTC m=+40.237833399"
Jul 16 17:35:14.685282 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:14.685193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:16.736238 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:16.736189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:16.803926 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:16.803898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:16.899634 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:16.899608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:16.899750 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:16.899659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:16.899750 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:16.899745    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:16.899837 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:16.899799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:24.899786342 +0000 UTC m=+51.656194381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:16.899881 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:16.899841    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:16.899881 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:16.899858    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:16.899936 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:16.899908    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:24.899892444 +0000 UTC m=+51.656300506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:19.281469 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:19.281439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:19.393851 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:19.393820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:20.672153 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:20.672124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:21.502012 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:21.501956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:21.601121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:21.601089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:21.706625 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:21.706593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:24.955521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:24.955476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:24.955547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:24.955641    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:24.955683    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:24.955700    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:24.955728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:40.955706474 +0000 UTC m=+67.712114537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:24.955900 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:24.955744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:40.955736698 +0000 UTC m=+67.712144739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:27.973827 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:27.973792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:28.059551 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:28.059486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:31.524589 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:31.524554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:31.621961 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:31.621930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:33.241914 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:33.241880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:33.353097 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:33.353069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:36.442000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:36.441974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhthf"
Jul 16 17:35:36.643984 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:36.643958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:36.728838 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:36.728779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:39.054602 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.054558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:39.055012 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.054646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:35:39.059579 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.059561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:39.059626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.059595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:39.065514 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:39.065484    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 17:35:39.065604 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:39.065590    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs podName:9e650ce2-5cf1-4786-8905-9c4775e5c64c nodeName:}" failed. No retries permitted until 2026-07-16 17:36:43.065568698 +0000 UTC m=+129.821976742 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs") pod "network-metrics-daemon-tgz85" (UID: "9e650ce2-5cf1-4786-8905-9c4775e5c64c") : secret "metrics-daemon-secret" not found
Jul 16 17:35:39.069623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.069590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e6da779-aad7-4536-a3aa-30ae688b10a4-original-pull-secret\") pod \"global-pull-secret-syncer-785gt\" (UID: \"4e6da779-aad7-4536-a3aa-30ae688b10a4\") " pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:39.155551 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.155515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:39.158643 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.158625    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:39.171655 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.171633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:39.178224 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.178206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhbw\" (UniqueName: \"kubernetes.io/projected/5c6175b9-2bb5-4715-a352-d0a7db815981-kube-api-access-jwhbw\") pod \"network-check-target-9tvfn\" (UID: \"5c6175b9-2bb5-4715-a352-d0a7db815981\") " pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:39.314377 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.314305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28hgh\""
Jul 16 17:35:39.315368 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.315350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785gt"
Jul 16 17:35:39.321655 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.321637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:39.477806 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.477778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-785gt"]
Jul 16 17:35:39.481086 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:35:39.481064    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6da779_aad7_4536_a3aa_30ae688b10a4.slice/crio-ac5ecf3df1fe630406c536ca370b012886ca1fd5ab511832dcadc345f3e4ea0d WatchSource:0}: Error finding container ac5ecf3df1fe630406c536ca370b012886ca1fd5ab511832dcadc345f3e4ea0d: Status 404 returned error can't find the container with id ac5ecf3df1fe630406c536ca370b012886ca1fd5ab511832dcadc345f3e4ea0d
Jul 16 17:35:39.496253 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.496231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9tvfn"]
Jul 16 17:35:39.499004 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:35:39.498984    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6175b9_2bb5_4715_a352_d0a7db815981.slice/crio-4562a84bda32e078aebdadcac387412c799c7e648027b3ec8cfdfe00e1894975 WatchSource:0}: Error finding container 4562a84bda32e078aebdadcac387412c799c7e648027b3ec8cfdfe00e1894975: Status 404 returned error can't find the container with id 4562a84bda32e078aebdadcac387412c799c7e648027b3ec8cfdfe00e1894975
Jul 16 17:35:39.514767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.514737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9tvfn" event={"ID":"5c6175b9-2bb5-4715-a352-d0a7db815981","Type":"ContainerStarted","Data":"4562a84bda32e078aebdadcac387412c799c7e648027b3ec8cfdfe00e1894975"}
Jul 16 17:35:39.515752 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:39.515731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-785gt" event={"ID":"4e6da779-aad7-4536-a3aa-30ae688b10a4","Type":"ContainerStarted","Data":"ac5ecf3df1fe630406c536ca370b012886ca1fd5ab511832dcadc345f3e4ea0d"}
Jul 16 17:35:40.950183 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:40.950145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:40.966208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:40.966177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:35:40.966340 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:40.966233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:35:40.966391 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:40.966342    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:40.966391 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:40.966385    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:40.966483 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:40.966399    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:35:40.966483 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:40.966412    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.966392915 +0000 UTC m=+99.722800959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:40.966483 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:35:40.966445    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.966429872 +0000 UTC m=+99.722837916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:35:42.038772 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:42.038737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:44.530774 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:44.530741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9tvfn" event={"ID":"5c6175b9-2bb5-4715-a352-d0a7db815981","Type":"ContainerStarted","Data":"7f3254de3b991930f7f3d1b4bcc54b18fa17322d1a5feb4ecf157fb78c273b85"}
Jul 16 17:35:44.531154 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:44.530870    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:35:44.532281 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:44.532261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-785gt" event={"ID":"4e6da779-aad7-4536-a3aa-30ae688b10a4","Type":"ContainerStarted","Data":"cf6643341a1b63305f5999f91d7757ba541b8e5ddd89f2ce5859a6db75a2eda5"}
Jul 16 17:35:44.592735 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:44.592695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-785gt" podStartSLOduration=64.868275082 podStartE2EDuration="1m9.592681709s" totalImagesPullingTime="4.724406627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:44.591008276 +0000 UTC m=+71.347416374" watchObservedRunningTime="2026-07-16 17:35:44.592681709 +0000 UTC m=+71.349089772"
Jul 16 17:35:44.593089 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:44.593070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9tvfn" podStartSLOduration=64.899175134 podStartE2EDuration="1m9.593063368s" totalImagesPullingTime="4.693888234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:44.559122808 +0000 UTC m=+71.315530869" watchObservedRunningTime="2026-07-16 17:35:44.593063368 +0000 UTC m=+71.349471408"
Jul 16 17:35:51.223842 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:51.223804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:51.311116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:51.311063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:53.513702 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:53.513656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:35:53.593883 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:35:53.593847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:01.032043 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.031901    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-psmpz"]
Jul 16 17:36:01.034867 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.034846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.041301 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.041273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 17:36:01.041444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.041318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 17:36:01.041444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.041393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 17:36:01.042373 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.041836    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 17:36:01.043664 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.043646    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2dqzc\""
Jul 16 17:36:01.051366 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.048237    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 17:36:01.093187 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-tmp\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.093299 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-serving-cert\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.093299 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.093397 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.093444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-snapshots\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.093444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.093438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ft7c\" (UniqueName: \"kubernetes.io/projected/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-kube-api-access-6ft7c\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.167635 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.167616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-psmpz"]
Jul 16 17:36:01.194010 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.193990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-snapshots\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194103 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ft7c\" (UniqueName: \"kubernetes.io/projected/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-kube-api-access-6ft7c\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194103 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-tmp\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194103 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-serving-cert\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194103 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194309 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194609 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-snapshots\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194609 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-tmp\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.194716 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.194705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.195034 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.195017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.196460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.196446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-serving-cert\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.213882 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.213865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ft7c\" (UniqueName: \"kubernetes.io/projected/49b1cdcc-0bd7-491b-bac9-26b8e763a2dd-kube-api-access-6ft7c\") pod \"insights-operator-68d9dd94cf-psmpz\" (UID: \"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd\") " pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.345579 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.345529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz"
Jul 16 17:36:01.480257 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.480230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-psmpz"]
Jul 16 17:36:01.483361 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:01.483330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b1cdcc_0bd7_491b_bac9_26b8e763a2dd.slice/crio-c3dfc4556b43e7f1fa20617668b69e24eeca4cfcc34b8c88f350769df4993313 WatchSource:0}: Error finding container c3dfc4556b43e7f1fa20617668b69e24eeca4cfcc34b8c88f350769df4993313: Status 404 returned error can't find the container with id c3dfc4556b43e7f1fa20617668b69e24eeca4cfcc34b8c88f350769df4993313
Jul 16 17:36:01.573769 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:01.573737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz" event={"ID":"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd","Type":"ContainerStarted","Data":"c3dfc4556b43e7f1fa20617668b69e24eeca4cfcc34b8c88f350769df4993313"}
Jul 16 17:36:03.580923 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:03.580889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz" event={"ID":"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd","Type":"ContainerStarted","Data":"377cd82da5938d876cb4f5ff24aa0f201aaf1b12ed597cd5858e2b1e085b6b42"}
Jul 16 17:36:03.604136 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:03.604094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz" podStartSLOduration=0.70910177 podStartE2EDuration="2.604078687s" totalImagesPullingTime="1.894976917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:03.602947036 +0000 UTC m=+90.359355098" watchObservedRunningTime="2026-07-16 17:36:03.604078687 +0000 UTC m=+90.360486766"
Jul 16 17:36:05.030312 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.030283    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l9zbv"]
Jul 16 17:36:05.032461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.032446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.036078 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.036054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnl5b\""
Jul 16 17:36:05.036305 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.036285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 17:36:05.036765 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.036749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 17:36:05.046196 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.046174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9zbv"]
Jul 16 17:36:05.058726 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.058706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:05.121742 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.121716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa3c374-845e-4279-9205-9cf5bd8597fa-crio-socket\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.121845 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.121750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.121845 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.121768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b64l\" (UniqueName: \"kubernetes.io/projected/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-api-access-2b64l\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.121938 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.121848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa3c374-845e-4279-9205-9cf5bd8597fa-data-volume\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.121938 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.121880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa3c374-845e-4279-9205-9cf5bd8597fa-crio-socket\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b64l\" (UniqueName: \"kubernetes.io/projected/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-api-access-2b64l\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa3c374-845e-4279-9205-9cf5bd8597fa-data-volume\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223384 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa3c374-845e-4279-9205-9cf5bd8597fa-crio-socket\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.223384 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.223203    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.223384 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.223240    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.223384 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.223272    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:05.723256607 +0000 UTC m=+92.479664663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.223384 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.223286    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:05.723279874 +0000 UTC m=+92.479687915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.223617 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.223426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa3c374-845e-4279-9205-9cf5bd8597fa-data-volume\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.236188 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.236164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b64l\" (UniqueName: \"kubernetes.io/projected/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-api-access-2b64l\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.725406 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.725369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.725611 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:05.725429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:05.725611 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.725520    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.725611 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.725570    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.725746 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.725580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.725565735 +0000 UTC m=+93.481973775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.725746 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:05.725646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.725628846 +0000 UTC m=+93.482036903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:06.685360 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:06.685331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:06.733659 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:06.733632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:06.733755 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:06.733702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:06.733789 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:06.733754    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:06.733826 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:06.733798    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:06.733858 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:06.733826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:08.733808322 +0000 UTC m=+95.490216373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:06.733858 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:06.733841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:08.733834589 +0000 UTC m=+95.490242629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:06.777982 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:06.777956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.235326 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.235298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nfsf7_658633e6-f491-4234-b88e-482a65cdab95/dns-node-resolver/0.log"
Jul 16 17:36:07.288098 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.288065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.385682 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.385650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.452157 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.452126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.593288 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.593252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.761153 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.761122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:07.814286 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.814263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fq5kb_3b8f3865-fae0-4b1b-8a8f-92f8776798b6/node-ca/0.log"
Jul 16 17:36:07.831715 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:07.831694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:08.617161 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:08.617134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dmmld_0a3dce21-336b-42fb-81b0-59104d77c969/serve-healthcheck-canary/0.log"
Jul 16 17:36:08.745213 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:08.745179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:08.745374 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:08.745232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:08.745374 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:08.745315    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:08.745374 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:08.745328    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:08.745481 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:08.745378    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.745361545 +0000 UTC m=+99.501769588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:08.745481 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:08.745393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.745386748 +0000 UTC m=+99.501794788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:08.855051 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:08.855013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:08.950553 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:08.950452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:10.877637 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:10.877596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:10.979298 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:10.979266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:11.278958 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:11.278921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.092635 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.092602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.213561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.213511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.315611 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.315574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.496949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.496925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.587213 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.587188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.688815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.688796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.770957 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.770905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:12.771035 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.770956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:12.771080 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.771053    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:12.771119 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.771109    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:20.771096988 +0000 UTC m=+107.527505032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:12.771168 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.771052    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:12.771203 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.771178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:20.771166462 +0000 UTC m=+107.527574502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:12.812474 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.812453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:12.973004 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.972976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:36:12.973090 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:12.973026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") pod \"image-registry-7cb59898df-5dfdd\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") " pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:36:12.973128 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.973114    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:36:12.973183 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.973173    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls podName:c5885fde-e5b3-423c-9442-990136b5a521 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:16.973156273 +0000 UTC m=+163.729564312 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls") pod "dns-default-j6rxh" (UID: "c5885fde-e5b3-423c-9442-990136b5a521") : secret "dns-default-metrics-tls" not found
Jul 16 17:36:12.973307 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.973292    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:36:12.973340 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.973311    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb59898df-5dfdd: secret "image-registry-tls" not found
Jul 16 17:36:12.973411 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:12.973362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls podName:2ddfe9f5-bbcd-40bb-a614-313f2cf11646 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:16.973346229 +0000 UTC m=+163.729754283 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls") pod "image-registry-7cb59898df-5dfdd" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646") : secret "image-registry-tls" not found
Jul 16 17:36:13.968394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:13.968344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.160745 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.160720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.369744 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.369708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.585019 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.584993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.650780 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.650724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.768284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.768247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:14.849441 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:14.849422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:15.538343 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:15.538319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9tvfn"
Jul 16 17:36:15.806081 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:15.806016    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:15.894126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:15.894105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:15.989445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:15.989420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:16.176158 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:16.176096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:16.367437 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:16.367407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:16.573030 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:16.573010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:16.671100 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:16.671074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:16.786829 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:16.786805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:17.772417 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:17.772384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:18.164298 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:18.164213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:18.267192 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:18.267158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:18.786192 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:18.786159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:18.842326 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:18.842300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:19.968600 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:19.968564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:20.166765 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.166738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:20.361894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.361867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:20.562493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.562462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:20.773751 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.773722    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:20.829436 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.829414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:20.829575 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.829459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:20.829639 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:20.829590    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:20.829687 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:20.829664    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls podName:eaa3c374-845e-4279-9205-9cf5bd8597fa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:36.829642293 +0000 UTC m=+123.586050335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9zbv" (UID: "eaa3c374-845e-4279-9205-9cf5bd8597fa") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:20.829935 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:20.829916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa3c374-845e-4279-9205-9cf5bd8597fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:21.585609 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:21.585577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:22.167526 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:22.167476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:22.381457 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:22.381424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:22.584694 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:22.584660    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:22.767525 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:22.767472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:23.100444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:23.100408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:24.841143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:24.841109    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:24.941461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:24.941439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:25.414279 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:25.414241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:25.514803 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:25.514767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:25.575722 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:25.575691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:25.712476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:25.712381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:25.812473 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:25.812449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:26.518276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:26.518239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:26.620260 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:26.620205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:26.720648 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:26.720622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:26.821908 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:26.821847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:26.922755 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:26.922734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:27.022175 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:27.022129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:27.628809 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:27.628775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:27.682932 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:27.682900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:31.254410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.254374    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"]
Jul 16 17:36:31.256688 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.256668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.260729 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.260705    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 17:36:31.273299 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.273279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 17:36:31.273401 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.273381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 17:36:31.273448 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.273380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 17:36:31.283164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.283142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 17:36:31.287056 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.287036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 17:36:31.287430 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.287400    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 17:36:31.288191 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.288169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"]
Jul 16 17:36:31.300972 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.300951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.301083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.300976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.301083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.301005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2q97\" (UniqueName: \"kubernetes.io/projected/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-kube-api-access-f2q97\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.301163 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.301103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.301163 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.301131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.301163 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.301149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401641 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2q97\" (UniqueName: \"kubernetes.io/projected/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-kube-api-access-f2q97\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401844 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.401844 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.401767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.402485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.402463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.404075 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.404048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-ca\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.404177 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.404094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.404249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.404185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.404298 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.404251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.413560 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.413496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:31.426394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.426367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2q97\" (UniqueName: \"kubernetes.io/projected/1c0b4666-5599-4ae8-9f75-a0ba05e22f27-kube-api-access-f2q97\") pod \"cluster-proxy-proxy-agent-56669554d4-x5n9d\" (UID: \"1c0b4666-5599-4ae8-9f75-a0ba05e22f27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.505491 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.505430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:31.574545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.574519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"
Jul 16 17:36:31.603890 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.603863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:31.713695 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.713660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d"]
Jul 16 17:36:31.715716 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:31.715693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c0b4666_5599_4ae8_9f75_a0ba05e22f27.slice/crio-6ad4b97873c9e33966baaa33d1614e1b9e2027ef9cd6802d59c43f9b0315bada WatchSource:0}: Error finding container 6ad4b97873c9e33966baaa33d1614e1b9e2027ef9cd6802d59c43f9b0315bada: Status 404 returned error can't find the container with id 6ad4b97873c9e33966baaa33d1614e1b9e2027ef9cd6802d59c43f9b0315bada
Jul 16 17:36:31.720728 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.720705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:31.819740 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:31.819690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:32.650786 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:32.650731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d" event={"ID":"1c0b4666-5599-4ae8-9f75-a0ba05e22f27","Type":"ContainerStarted","Data":"6ad4b97873c9e33966baaa33d1614e1b9e2027ef9cd6802d59c43f9b0315bada"}
Jul 16 17:36:35.660806 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.660762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d" event={"ID":"1c0b4666-5599-4ae8-9f75-a0ba05e22f27","Type":"ContainerStarted","Data":"f48f5b81662f22fa859715bc022768cff82c97b94e1ad89e1a55b95dacdaf7cb"}
Jul 16 17:36:35.661167 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.660814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d" event={"ID":"1c0b4666-5599-4ae8-9f75-a0ba05e22f27","Type":"ContainerStarted","Data":"eeec665671dea68978de6a75eb22cb1dd0c2492606a1d08afcd9c9f7906157b9"}
Jul 16 17:36:35.661167 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.660833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d" event={"ID":"1c0b4666-5599-4ae8-9f75-a0ba05e22f27","Type":"ContainerStarted","Data":"87f1b671f0b6d086e3853f0453d971839664d3a2e6de072556154f0248efa016"}
Jul 16 17:36:35.684384 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.684342    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56669554d4-x5n9d" podStartSLOduration=1.440394452 podStartE2EDuration="4.684328939s" totalImagesPullingTime="3.243934487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:35.683277175 +0000 UTC m=+122.439685248" watchObservedRunningTime="2026-07-16 17:36:35.684328939 +0000 UTC m=+122.440737000"
Jul 16 17:36:35.771066 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.771039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:35.861960 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.861927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:35.970327 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:35.970260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:36.783586 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:36.783553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:36.843852 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:36.843823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:36.846072 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:36.846046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa3c374-845e-4279-9205-9cf5bd8597fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9zbv\" (UID: \"eaa3c374-845e-4279-9205-9cf5bd8597fa\") " pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:36.882917 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:36.882887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:37.143149 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:37.143073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnl5b\""
Jul 16 17:36:37.150726 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:37.150694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9zbv"
Jul 16 17:36:37.277532 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:37.277494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9zbv"]
Jul 16 17:36:37.279413 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:37.279390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa3c374_845e_4279_9205_9cf5bd8597fa.slice/crio-d45a16cf231718d96414e9718a1968731173da04f7320b42a3e365ab2c83e766 WatchSource:0}: Error finding container d45a16cf231718d96414e9718a1968731173da04f7320b42a3e365ab2c83e766: Status 404 returned error can't find the container with id d45a16cf231718d96414e9718a1968731173da04f7320b42a3e365ab2c83e766
Jul 16 17:36:37.626091 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:37.626060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:37.667781 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:37.667752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9zbv" event={"ID":"eaa3c374-845e-4279-9205-9cf5bd8597fa","Type":"ContainerStarted","Data":"d45a16cf231718d96414e9718a1968731173da04f7320b42a3e365ab2c83e766"}
Jul 16 17:36:39.290545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:39.290490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:39.785570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:39.785523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:40.324330 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:40.324296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:41.000159 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.000103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:41.239418 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.239378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:41.336284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.336177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:41.368964 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.368936    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qnm77"]
Jul 16 17:36:41.384164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.384132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.390253 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.390228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 17:36:41.390584 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.390566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 17:36:41.391173 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.391153    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 17:36:41.396088 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.396070    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 17:36:41.399598 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.399579    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qkjzl\""
Jul 16 17:36:41.401062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.401044    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 17:36:41.414119 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.414096    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 17:36:41.479961 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.479933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-textfile\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.479970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-metrics-client-ca\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-wtmp\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-root\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdsr\" (UniqueName: \"kubernetes.io/projected/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-kube-api-access-hgdsr\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480344 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.480391 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.480342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-sys\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580759 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-wtmp\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580759 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-root\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-root\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-wtmp\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.580971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdsr\" (UniqueName: \"kubernetes.io/projected/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-kube-api-access-hgdsr\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.580993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-sys\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-textfile\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-metrics-client-ca\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-sys\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581212 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:41.581154    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 17:36:41.581615 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:41.581217    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls podName:8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f nodeName:}" failed. No retries permitted until 2026-07-16 17:36:42.081197002 +0000 UTC m=+128.837605046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls") pod "node-exporter-qnm77" (UID: "8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f") : secret "node-exporter-tls" not found
Jul 16 17:36:41.581615 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581615 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-textfile\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.581754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.581633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-metrics-client-ca\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.583457 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.583433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:41.599396 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:41.599344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdsr\" (UniqueName: \"kubernetes.io/projected/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-kube-api-access-hgdsr\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:42.086205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.086169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:42.088512 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.088481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f-node-exporter-tls\") pod \"node-exporter-qnm77\" (UID: \"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f\") " pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:42.295250 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.295214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qnm77"
Jul 16 17:36:42.455874 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.455806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:42.546786 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.546750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:42.864666 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.864633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:42.966741 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:42.966707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:43.097192 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.097158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:36:43.099775 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.099748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e650ce2-5cf1-4786-8905-9c4775e5c64c-metrics-certs\") pod \"network-metrics-daemon-tgz85\" (UID: \"9e650ce2-5cf1-4786-8905-9c4775e5c64c\") " pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:36:43.310741 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.310703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4hnr\""
Jul 16 17:36:43.318882 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.318860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgz85"
Jul 16 17:36:43.456930 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:43.456896    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d99c32a_04f2_4678_ad1f_dba4d9ec7f4f.slice/crio-2a35085909ac78701e927fc5ad1ecca77cdfe57545cd1d0fe8245cc38bd9a196 WatchSource:0}: Error finding container 2a35085909ac78701e927fc5ad1ecca77cdfe57545cd1d0fe8245cc38bd9a196: Status 404 returned error can't find the container with id 2a35085909ac78701e927fc5ad1ecca77cdfe57545cd1d0fe8245cc38bd9a196
Jul 16 17:36:43.582767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.582745    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgz85"]
Jul 16 17:36:43.585428 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:43.585398    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e650ce2_5cf1_4786_8905_9c4775e5c64c.slice/crio-df565451a7e0d859d187f3a98f176768d5d45241f0e83f29604a3c594c78340b WatchSource:0}: Error finding container df565451a7e0d859d187f3a98f176768d5d45241f0e83f29604a3c594c78340b: Status 404 returned error can't find the container with id df565451a7e0d859d187f3a98f176768d5d45241f0e83f29604a3c594c78340b
Jul 16 17:36:43.686631 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.686599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9zbv" event={"ID":"eaa3c374-845e-4279-9205-9cf5bd8597fa","Type":"ContainerStarted","Data":"4f30cfa8c27f8fc316560ce3de837220fa11275ff9e88fcab611f116b9bfd966"}
Jul 16 17:36:43.687698 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.687673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgz85" event={"ID":"9e650ce2-5cf1-4786-8905-9c4775e5c64c","Type":"ContainerStarted","Data":"df565451a7e0d859d187f3a98f176768d5d45241f0e83f29604a3c594c78340b"}
Jul 16 17:36:43.688699 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.688680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnm77" event={"ID":"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f","Type":"ContainerStarted","Data":"2a35085909ac78701e927fc5ad1ecca77cdfe57545cd1d0fe8245cc38bd9a196"}
Jul 16 17:36:43.759953 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:43.759925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:44.353352 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:44.353311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:44.476228 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:44.476193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:44.574219 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:44.574186    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:44.801982 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:44.801945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:45.971604 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:45.971573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:46.077388 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.077361    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6587bc5797-tkxtf"]
Jul 16 17:36:46.087447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.087423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.091237 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.091220    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 17:36:46.091412 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.091386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 17:36:46.091733 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.091713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 17:36:46.092593 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.092578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7mt5vktm5cpml\""
Jul 16 17:36:46.099769 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.099743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 17:36:46.099882 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.099863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cv75n\""
Jul 16 17:36:46.101183 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.101163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6587bc5797-tkxtf"]
Jul 16 17:36:46.148828 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.148805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:46.221706 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3496c976-cb64-46fa-8987-3fefef17ac59-audit-log\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.221706 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-client-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.221885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-tls\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.221885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbq8\" (UniqueName: \"kubernetes.io/projected/3496c976-cb64-46fa-8987-3fefef17ac59-kube-api-access-zzbq8\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.221885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-client-certs\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.222015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.222015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.221940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-metrics-server-audit-profiles\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.277455 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.277419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:46.323100 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-tls\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323100 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbq8\" (UniqueName: \"kubernetes.io/projected/3496c976-cb64-46fa-8987-3fefef17ac59-kube-api-access-zzbq8\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-client-certs\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-metrics-server-audit-profiles\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323437 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3496c976-cb64-46fa-8987-3fefef17ac59-audit-log\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.323437 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.323347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-client-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.324060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.324034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3496c976-cb64-46fa-8987-3fefef17ac59-audit-log\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.324236 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.324215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.324480 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.324454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3496c976-cb64-46fa-8987-3fefef17ac59-metrics-server-audit-profiles\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.326016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.325977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-tls\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.326147 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.326123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-client-ca-bundle\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.326224 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.326128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3496c976-cb64-46fa-8987-3fefef17ac59-secret-metrics-server-client-certs\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.334074 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.334055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbq8\" (UniqueName: \"kubernetes.io/projected/3496c976-cb64-46fa-8987-3fefef17ac59-kube-api-access-zzbq8\") pod \"metrics-server-6587bc5797-tkxtf\" (UID: \"3496c976-cb64-46fa-8987-3fefef17ac59\") " pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:46.382913 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.382877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:46.397851 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:46.397825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:36:47.660099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.660038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:47.702812 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.702787    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f" containerID="61b77d74b7f00e2cadaa2d0a90096cd8dfc51b92750ef1981d70531948979a69" exitCode=0
Jul 16 17:36:47.702926 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.702872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnm77" event={"ID":"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f","Type":"ContainerDied","Data":"61b77d74b7f00e2cadaa2d0a90096cd8dfc51b92750ef1981d70531948979a69"}
Jul 16 17:36:47.704682 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.704656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9zbv" event={"ID":"eaa3c374-845e-4279-9205-9cf5bd8597fa","Type":"ContainerStarted","Data":"44fa0f31d232f1d279ea797b7355a120ed59ae7ee9a6fdad8ff3442a6d297b20"}
Jul 16 17:36:47.706410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.706390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgz85" event={"ID":"9e650ce2-5cf1-4786-8905-9c4775e5c64c","Type":"ContainerStarted","Data":"2fcd80deda5ed87e35b8f323e7b7c7e74743f5a066b07be2431f9aeddcba289c"}
Jul 16 17:36:47.714077 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:47.713735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6587bc5797-tkxtf"]
Jul 16 17:36:47.744439 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:47.744381    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496c976_cb64_46fa_8987_3fefef17ac59.slice/crio-ade2de782689bc8bf83c8ecf25c942c8888746d99d1a355202c946470c1dec1f WatchSource:0}: Error finding container ade2de782689bc8bf83c8ecf25c942c8888746d99d1a355202c946470c1dec1f: Status 404 returned error can't find the container with id ade2de782689bc8bf83c8ecf25c942c8888746d99d1a355202c946470c1dec1f
Jul 16 17:36:48.037885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.037810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:36:48.041242 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.041220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.050282 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 17:36:48.050455 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 17:36:48.050497 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 17:36:48.050639 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 17:36:48.050919 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mxfsz\""
Jul 16 17:36:48.050978 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 17:36:48.051010 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.050967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 17:36:48.052114 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.052101    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 17:36:48.053715 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.053703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 17:36:48.053814 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.053796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 17:36:48.053876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.053827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ttnp7rhppg11\""
Jul 16 17:36:48.053944 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.053926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 17:36:48.054643 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.054630    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 17:36:48.055802 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.055784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 17:36:48.056631 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.056617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 17:36:48.065815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.065795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 17:36:48.072542 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.072519    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:48.140155 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140155 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbz4w\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.140734 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.141116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.141116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.141116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.141116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.140881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.161417 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.161385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:36:48.241858 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.241858 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.241983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242099 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbz4w\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.242988 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.242464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.243112 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.243089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.244454 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.244063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.244454 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.244341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.245068 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.245009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.246875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.246847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.247064 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.247023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.247193 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.247152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.247960 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.247932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.249827 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.249787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.249986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250353 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250417 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250895 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250895 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.250895 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.251088 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.250954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.252171 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.252114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.254892 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.254836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbz4w\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w\") pod \"prometheus-k8s-0\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.350888 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.350733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:36:48.702854 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.702810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:36:48.712846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.712763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgz85" event={"ID":"9e650ce2-5cf1-4786-8905-9c4775e5c64c","Type":"ContainerStarted","Data":"bb2c8a73d7c36de24860d78c45b4944b2598e3fff88b7e4735e45112dc1b4b07"}
Jul 16 17:36:48.715324 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.715268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnm77" event={"ID":"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f","Type":"ContainerStarted","Data":"fdc40bb920b712d7bccdfcfadffb85d45b095057a410aa0e8b1a6620ec81c88b"}
Jul 16 17:36:48.715324 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.715313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnm77" event={"ID":"8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f","Type":"ContainerStarted","Data":"7934940f58ac1968562b459746084276b2737458c9f8a9c36f0ec4dcfa6f27f0"}
Jul 16 17:36:48.716813 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.716780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf" event={"ID":"3496c976-cb64-46fa-8987-3fefef17ac59","Type":"ContainerStarted","Data":"ade2de782689bc8bf83c8ecf25c942c8888746d99d1a355202c946470c1dec1f"}
Jul 16 17:36:48.752967 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.752922    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tgz85" podStartSLOduration=129.820831199 podStartE2EDuration="2m13.752906136s" totalImagesPullingTime="3.932074937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:48.752139908 +0000 UTC m=+135.508547971" watchObservedRunningTime="2026-07-16 17:36:48.752906136 +0000 UTC m=+135.509314197"
Jul 16 17:36:48.787661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:48.787611    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qnm77" podStartSLOduration=3.648503792 podStartE2EDuration="7.787593278s" totalImagesPullingTime="4.059971972s" totalInitContainerRuntime="79.117514ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:48.786596077 +0000 UTC m=+135.543004146" watchObservedRunningTime="2026-07-16 17:36:48.787593278 +0000 UTC m=+135.544001340"
Jul 16 17:36:49.096968 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:36:49.096937    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b04f0_9768_45ca_ad33_160357f4503f.slice/crio-da9afa80549b849db7770f8652e77b591e221b30ffb5b911d1d8be5be5483554 WatchSource:0}: Error finding container da9afa80549b849db7770f8652e77b591e221b30ffb5b911d1d8be5be5483554: Status 404 returned error can't find the container with id da9afa80549b849db7770f8652e77b591e221b30ffb5b911d1d8be5be5483554
Jul 16 17:36:49.723633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:49.723595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf" event={"ID":"3496c976-cb64-46fa-8987-3fefef17ac59","Type":"ContainerStarted","Data":"416ef4d91fcda5fb457d7a6bb1d8dba021675a9ad76cf628c5b4950a26d28550"}
Jul 16 17:36:49.725203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:49.725154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"da9afa80549b849db7770f8652e77b591e221b30ffb5b911d1d8be5be5483554"}
Jul 16 17:36:49.734970 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:49.734944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:49.827340 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:49.827259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:51.736567 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:51.736536    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488" exitCode=0
Jul 16 17:36:51.736942 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:51.736620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"}
Jul 16 17:36:51.738733 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:51.738711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9zbv" event={"ID":"eaa3c374-845e-4279-9205-9cf5bd8597fa","Type":"ContainerStarted","Data":"32a0c11e2e6fa6dc1dfe34f753eece9353695ba568375c776df49d8922a8b0a0"}
Jul 16 17:36:51.777588 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:51.777540    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf" podStartSLOduration=4.379763592 podStartE2EDuration="5.777526631s" totalImagesPullingTime="1.397763039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:49.752308997 +0000 UTC m=+136.508717061" watchObservedRunningTime="2026-07-16 17:36:51.777526631 +0000 UTC m=+138.533934692"
Jul 16 17:36:51.813721 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:51.813650    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l9zbv" podStartSLOduration=32.662184549 podStartE2EDuration="46.813636972s" totalImagesPullingTime="14.151452423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:51.813169301 +0000 UTC m=+138.569577363" watchObservedRunningTime="2026-07-16 17:36:51.813636972 +0000 UTC m=+138.570045033"
Jul 16 17:36:53.020304 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:53.020272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:53.120664 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:53.120619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:53.903868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:53.903823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.011923 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.011866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.235259 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.235167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.336149 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.336119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.659145 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.659116    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb59898df-5dfdd"]
Jul 16 17:36:54.659676 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:36:54.659645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7cb59898df-5dfdd" podUID="2ddfe9f5-bbcd-40bb-a614-313f2cf11646"
Jul 16 17:36:54.752611 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.752584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:36:54.752825 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.752798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"}
Jul 16 17:36:54.752967 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.752874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"}
Jul 16 17:36:54.759995 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.759975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:36:54.790045 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.790016    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.873034 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.872996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:54.904010 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.903982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904157    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96pbv\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904231    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:36:54.904284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token\") pod \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\" (UID: \"2ddfe9f5-bbcd-40bb-a614-313f2cf11646\") "
Jul 16 17:36:54.904473 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:54.904553 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904498    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-ca-trust-extracted\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:54.904553 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904545    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-certificates\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:54.904672 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.904637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:54.906303 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.906277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:54.906380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.906359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:54.906415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.906359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv" (OuterVolumeSpecName: "kube-api-access-96pbv") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "kube-api-access-96pbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:54.906415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.906380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2ddfe9f5-bbcd-40bb-a614-313f2cf11646" (UID: "2ddfe9f5-bbcd-40bb-a614-313f2cf11646"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:54.978229 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:54.978206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:55.005595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.005547    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-installation-pull-secrets\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:55.005595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.005566    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-trusted-ca\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:55.005595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.005575    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96pbv\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-kube-api-access-96pbv\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:55.005595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.005583    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-bound-sa-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:55.005595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.005592    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-image-registry-private-configuration\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:55.084040 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.084006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:55.756248 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.756216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb59898df-5dfdd"
Jul 16 17:36:55.810421 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.810396    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb59898df-5dfdd"]
Jul 16 17:36:55.816916 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.816898    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cb59898df-5dfdd"]
Jul 16 17:36:55.911225 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:55.911205    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ddfe9f5-bbcd-40bb-a614-313f2cf11646-registry-tls\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:36:56.303700 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:56.303668    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ddfe9f5-bbcd-40bb-a614-313f2cf11646" path="/var/lib/kubelet/pods/2ddfe9f5-bbcd-40bb-a614-313f2cf11646/volumes"
Jul 16 17:36:56.850794 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:56.850755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:56.959133 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:56.959098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:36:57.766056 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:57.766026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"}
Jul 16 17:36:57.766151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:57.766093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"}
Jul 16 17:36:57.766151 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:57.766119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"}
Jul 16 17:36:58.777428 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:58.777391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerStarted","Data":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"}
Jul 16 17:36:58.817946 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:36:58.817891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.704760634 podStartE2EDuration="10.817871266s" totalImagesPullingTime="8.094621434s" totalInitContainerRuntime="18.489198ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:58.815699548 +0000 UTC m=+145.572107610" watchObservedRunningTime="2026-07-16 17:36:58.817871266 +0000 UTC m=+145.574279329"
Jul 16 17:37:01.680767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:01.680725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:01.783080 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:01.783053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:02.914740 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:02.914706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:03.012998 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:03.012969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:03.113343 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:03.113315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:03.350922 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:03.350895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:37:05.914741 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:05.914710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:06.016129 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:06.016093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:06.398585 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:06.398556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:37:06.398723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:06.398624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:37:09.140072 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:09.140039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:09.270756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:09.270722    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:09.374916 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:09.374892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:12.131413 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:37:12.131379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-j6rxh" podUID="c5885fde-e5b3-423c-9442-990136b5a521"
Jul 16 17:37:12.824276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:12.824246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:17.060806 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.060767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:17.063015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.062995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5885fde-e5b3-423c-9442-990136b5a521-metrics-tls\") pod \"dns-default-j6rxh\" (UID: \"c5885fde-e5b3-423c-9442-990136b5a521\") " pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:17.328199 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.328124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2d8t\""
Jul 16 17:37:17.336381 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.336356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:17.559517 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.559474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:17.615315 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.615241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:17.648264 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.648110    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6rxh"]
Jul 16 17:37:17.650346 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:37:17.650312    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5885fde_e5b3_423c_9442_990136b5a521.slice/crio-3bb44e798c8a99e776645816f6061a81112fdc5b240ceccbb58ce5e29647c925 WatchSource:0}: Error finding container 3bb44e798c8a99e776645816f6061a81112fdc5b240ceccbb58ce5e29647c925: Status 404 returned error can't find the container with id 3bb44e798c8a99e776645816f6061a81112fdc5b240ceccbb58ce5e29647c925
Jul 16 17:37:17.840961 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:17.840924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6rxh" event={"ID":"c5885fde-e5b3-423c-9442-990136b5a521","Type":"ContainerStarted","Data":"3bb44e798c8a99e776645816f6061a81112fdc5b240ceccbb58ce5e29647c925"}
Jul 16 17:37:20.854093 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:20.854056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6rxh" event={"ID":"c5885fde-e5b3-423c-9442-990136b5a521","Type":"ContainerStarted","Data":"8ffa32b7d40e2b4f9ba587d2bebb488777b889d58de4b1bfbb5e7bce4951f74b"}
Jul 16 17:37:20.854492 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:20.854108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6rxh" event={"ID":"c5885fde-e5b3-423c-9442-990136b5a521","Type":"ContainerStarted","Data":"b913c6793f7ba81cdde8bfcce173eb0c1c83e2e0f2f99b8f0fab41cde342d3d0"}
Jul 16 17:37:20.854492 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:20.854192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:20.875478 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:20.875416    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j6rxh" podStartSLOduration=129.788975475 podStartE2EDuration="2m11.875399173s" totalImagesPullingTime="2.086423698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:20.875232865 +0000 UTC m=+167.631640927" watchObservedRunningTime="2026-07-16 17:37:20.875399173 +0000 UTC m=+167.631807237"
Jul 16 17:37:26.403966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:26.403879    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:37:26.407865 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:26.407837    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6587bc5797-tkxtf"
Jul 16 17:37:29.886575 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:29.886541    2577 generic.go:356] "Generic (PLEG): container finished" podID="49b1cdcc-0bd7-491b-bac9-26b8e763a2dd" containerID="377cd82da5938d876cb4f5ff24aa0f201aaf1b12ed597cd5858e2b1e085b6b42" exitCode=0
Jul 16 17:37:29.886971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:29.886619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz" event={"ID":"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd","Type":"ContainerDied","Data":"377cd82da5938d876cb4f5ff24aa0f201aaf1b12ed597cd5858e2b1e085b6b42"}
Jul 16 17:37:29.886971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:29.886908    2577 scope.go:117] "RemoveContainer" containerID="377cd82da5938d876cb4f5ff24aa0f201aaf1b12ed597cd5858e2b1e085b6b42"
Jul 16 17:37:30.860166 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:30.860136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j6rxh"
Jul 16 17:37:30.891286 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:30.891246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-psmpz" event={"ID":"49b1cdcc-0bd7-491b-bac9-26b8e763a2dd","Type":"ContainerStarted","Data":"3a496382790d69cd9ba327d4748cdba32b0404e42207f1a0cd30cba29e1bbca0"}
Jul 16 17:37:40.109608 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:40.109570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:40.208899 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:40.208869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:37:48.351659 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:48.351627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:37:48.371133 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:48.371112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:37:48.967763 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:37:48.967712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:00.987620 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:00.987585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:01.339253 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:01.339220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:06.485826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.485795    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:06.486195 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486046    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="prometheus" containerID="cri-o://cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874" gracePeriod=600
Jul 16 17:38:06.486195 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486076    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy" containerID="cri-o://cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242" gracePeriod=600
Jul 16 17:38:06.486195 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486111    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7" gracePeriod=600
Jul 16 17:38:06.486195 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486142    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="thanos-sidecar" containerID="cri-o://a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e" gracePeriod=600
Jul 16 17:38:06.486195 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486148    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="config-reloader" containerID="cri-o://d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e" gracePeriod=600
Jul 16 17:38:06.486440 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.486131    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-web" containerID="cri-o://d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc" gracePeriod=600
Jul 16 17:38:06.725039 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.725015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:06.823487 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.823887 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbz4w\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823976    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.823997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"271b04f0-9768-45ca-ad33-160357f4503f\" (UID: \"271b04f0-9768-45ca-ad33-160357f4503f\") "
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.824274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824103    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.824748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824443    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.824748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824464    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.824748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824480    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.824894 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.824854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.825475 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.825434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.825624 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.825560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:38:06.826201 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.826174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.826304 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.826211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.826467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.826424    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:06.827597 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.827560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config" (OuterVolumeSpecName: "config") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.827597 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.827585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.827979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.827943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.827979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.827968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w" (OuterVolumeSpecName: "kube-api-access-dbz4w") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "kube-api-access-dbz4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:38:06.828152 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.828115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.828435 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.828401    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.828559 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.828442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out" (OuterVolumeSpecName: "config-out") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:38:06.828815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.828787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:38:06.828962 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.828945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.829404 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.829385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.836317 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.836297    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config" (OuterVolumeSpecName: "web-config") pod "271b04f0-9768-45ca-ad33-160357f4503f" (UID: "271b04f0-9768-45ca-ad33-160357f4503f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:06.925455 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925419    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925455 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925454    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-config-out\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925455 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925463    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-grpc-tls\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925472    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-kube-rbac-proxy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925482    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-configmap-metrics-client-ca\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925490    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-metrics-client-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925498    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbz4w\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-kube-api-access-dbz4w\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925522    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/271b04f0-9768-45ca-ad33-160357f4503f-tls-assets\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925531    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925538    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-db\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925567    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-web-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925576    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925584    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925593    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-config\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925602    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/271b04f0-9768-45ca-ad33-160357f4503f-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925611    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:06.925633 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:06.925620    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/271b04f0-9768-45ca-ad33-160357f4503f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:38:07.016203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016171    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7" exitCode=0
Jul 16 17:38:07.016203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016200    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242" exitCode=0
Jul 16 17:38:07.016203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016208    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc" exitCode=0
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016214    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e" exitCode=0
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016222    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e" exitCode=0
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016229    2577 generic.go:356] "Generic (PLEG): container finished" podID="271b04f0-9768-45ca-ad33-160357f4503f" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874" exitCode=0
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016243    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"}
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"}
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"}
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016443    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.016467 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"}
Jul 16 17:38:07.016876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"}
Jul 16 17:38:07.016876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"}
Jul 16 17:38:07.016876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.016610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"271b04f0-9768-45ca-ad33-160357f4503f","Type":"ContainerDied","Data":"da9afa80549b849db7770f8652e77b591e221b30ffb5b911d1d8be5be5483554"}
Jul 16 17:38:07.024402 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.024381    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.031787 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.031768    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.037696 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.037680    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.044071 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.044052    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.045203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.045179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:07.050648 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.050632    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.054850 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.054834    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:07.057346 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.057330    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.063359 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.063346    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.063610 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.063590    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.063654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.063617    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.063654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.063649    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.063837 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.063821    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.063875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.063841    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.063875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.063856    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.064066 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.064048    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.064129 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064074    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.064129 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064095    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.064336 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.064322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.064376 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064339    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.064376 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064351    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.064513 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.064488    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.064561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064519    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.064561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064531    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.064792 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.064774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.064829 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064797    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.064829 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.064812    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.065059 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:38:07.065042    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.065109 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065063    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.065109 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065078    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.065288 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065274    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.065335 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065288    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.065477 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065459    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.065528 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065478    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.065680 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065664    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.065723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065681    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.065867 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065852    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.065910 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.065870    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.066085 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066068    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.066085 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066084    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.066265 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066251    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.066265 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066264    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.066450 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066437    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.066496 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066449    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.066654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066640    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.066654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066652    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.066830 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066817    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.066876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066830    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.067006 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.066990    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.067006 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067005    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.067214 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067189    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.067214 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067207    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.067418 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067399    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.067468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067419    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.067629 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067614    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.067675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067630    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.067830 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067816    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.067866 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067830    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.068011 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.067996    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.068049 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068012    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.068206 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068189    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.068271 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068209    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.068421 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068403    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.068463 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068422    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.068656 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068639    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.068713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068656    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.068874 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068857    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.068936 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.068875    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.069075 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069057    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.069117 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069077    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.069284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069268    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.069338 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069284    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.069451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069432    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.069493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069452    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.069662 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069645    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.069662 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069661    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.069835 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069821    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.069835 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069834    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.070009 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.069993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.070047 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070009    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.070174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070158    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.070212 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070175    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.070341 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.070382 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070341    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.070533 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070514    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.070596 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070534    2577 scope.go:117] "RemoveContainer" containerID="05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"
Jul 16 17:38:07.070707 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070693    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7"} err="failed to get container status \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": rpc error: code = NotFound desc = could not find container \"05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7\": container with ID starting with 05aab08ce05ac85ca8642ff13860a6428cd3608476cfdec24c1d006b05ec37e7 not found: ID does not exist"
Jul 16 17:38:07.070745 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070707    2577 scope.go:117] "RemoveContainer" containerID="cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"
Jul 16 17:38:07.070873 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070857    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242"} err="failed to get container status \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": rpc error: code = NotFound desc = could not find container \"cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242\": container with ID starting with cdd2964ce484cb4e39112b3b13439da53e92d4df170a5a20e65ce9ef479eb242 not found: ID does not exist"
Jul 16 17:38:07.070923 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.070873    2577 scope.go:117] "RemoveContainer" containerID="d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"
Jul 16 17:38:07.071043 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071026    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc"} err="failed to get container status \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": rpc error: code = NotFound desc = could not find container \"d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc\": container with ID starting with d070b56ba2eaa9ec37a206941c38bff496fc2594479a0a461717730002b4b6fc not found: ID does not exist"
Jul 16 17:38:07.071043 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071042    2577 scope.go:117] "RemoveContainer" containerID="a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"
Jul 16 17:38:07.071240 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071223    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e"} err="failed to get container status \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": rpc error: code = NotFound desc = could not find container \"a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e\": container with ID starting with a26c2c9c253baf7bbecb38faf57994f570bb8abeb61e758dc6db07dc18f4967e not found: ID does not exist"
Jul 16 17:38:07.071310 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071241    2577 scope.go:117] "RemoveContainer" containerID="d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"
Jul 16 17:38:07.071461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071446    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e"} err="failed to get container status \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": rpc error: code = NotFound desc = could not find container \"d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e\": container with ID starting with d7a1bdbef7053390b28634ea0fec5677a62b854df26398c0194f05903325851e not found: ID does not exist"
Jul 16 17:38:07.071461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071460    2577 scope.go:117] "RemoveContainer" containerID="cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"
Jul 16 17:38:07.071646 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071632    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874"} err="failed to get container status \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": rpc error: code = NotFound desc = could not find container \"cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874\": container with ID starting with cce3c7c2a98bc5db3c916d1134e9f421ab4402e12db0abd606bcee3fa7af9874 not found: ID does not exist"
Jul 16 17:38:07.071694 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071646    2577 scope.go:117] "RemoveContainer" containerID="996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"
Jul 16 17:38:07.071832 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.071816    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488"} err="failed to get container status \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": rpc error: code = NotFound desc = could not find container \"996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488\": container with ID starting with 996d277ea564f84f09a2d1ccb9066ad0dd4b2266d259b144a90af6a835d56488 not found: ID does not exist"
Jul 16 17:38:07.088249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088197    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:07.088406 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088396    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="prometheus"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088407    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="prometheus"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088415    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="config-reloader"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="config-reloader"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088427    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-web"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088433    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-web"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="init-config-reloader"
Jul 16 17:38:07.088447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088445    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="init-config-reloader"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088451    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088456    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088463    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-thanos"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088469    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-thanos"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="thanos-sidecar"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088486    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="thanos-sidecar"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088541    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="prometheus"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088548    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-thanos"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088554    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="config-reloader"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088560    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="thanos-sidecar"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088565    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy"
Jul 16 17:38:07.088665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.088570    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b04f0-9768-45ca-ad33-160357f4503f" containerName="kube-rbac-proxy-web"
Jul 16 17:38:07.093423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.093407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.096796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.096782    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 17:38:07.096796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.096790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 17:38:07.098316 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.098293    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 17:38:07.098394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.098354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 17:38:07.098394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.098364    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 17:38:07.098394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.098369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 17:38:07.098685 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.098666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 17:38:07.101231 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 17:38:07.101231 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 17:38:07.101383 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 17:38:07.101545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 17:38:07.101609 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 17:38:07.101670 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101629    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ttnp7rhppg11\""
Jul 16 17:38:07.101876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.101861    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mxfsz\""
Jul 16 17:38:07.103293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.103274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 17:38:07.106665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.106650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 17:38:07.119834 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.119814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:07.228975 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.228922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.228996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229134 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wfz\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-kube-api-access-w9wfz\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229470 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229786 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.229786 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.229536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330385 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330385 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wfz\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-kube-api-access-w9wfz\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.330853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.330999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.331032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.331153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.331561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.331321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.332251 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.331939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.332375 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.332329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.332885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.332854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.333079 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.333054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.334580 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.334086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.334580 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.334286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.334709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.334605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2a753b5-db7a-4438-9e1b-66e0730df992-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335107 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335551 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2a753b5-db7a-4438-9e1b-66e0730df992-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.335798 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.335764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.336181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.336163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.336306 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.336292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.337060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.337042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2a753b5-db7a-4438-9e1b-66e0730df992-config\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.344464 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.344415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wfz\" (UniqueName: \"kubernetes.io/projected/f2a753b5-db7a-4438-9e1b-66e0730df992-kube-api-access-w9wfz\") pod \"prometheus-k8s-0\" (UID: \"f2a753b5-db7a-4438-9e1b-66e0730df992\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.402434 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.402400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:07.553636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:07.553600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 17:38:07.556440 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:38:07.556411    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a753b5_db7a_4438_9e1b_66e0730df992.slice/crio-ff698ad38f20954e418ff60e7a11edf55961c757d97c69448b33f990eaa35c5d WatchSource:0}: Error finding container ff698ad38f20954e418ff60e7a11edf55961c757d97c69448b33f990eaa35c5d: Status 404 returned error can't find the container with id ff698ad38f20954e418ff60e7a11edf55961c757d97c69448b33f990eaa35c5d
Jul 16 17:38:08.021715 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:08.021678    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2a753b5-db7a-4438-9e1b-66e0730df992" containerID="b775e870edf98490f106a2613dafaeb2dbc0ecb3b3bba2afee76443f0df80d01" exitCode=0
Jul 16 17:38:08.021872 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:08.021749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerDied","Data":"b775e870edf98490f106a2613dafaeb2dbc0ecb3b3bba2afee76443f0df80d01"}
Jul 16 17:38:08.021872 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:08.021813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"ff698ad38f20954e418ff60e7a11edf55961c757d97c69448b33f990eaa35c5d"}
Jul 16 17:38:08.305206 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:08.305180    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271b04f0-9768-45ca-ad33-160357f4503f" path="/var/lib/kubelet/pods/271b04f0-9768-45ca-ad33-160357f4503f/volumes"
Jul 16 17:38:09.028902 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.028872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"2da50be2594f88039f64ffa37147ecd7a63dc71133114b419fe013a32f5df409"}
Jul 16 17:38:09.029241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.028935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"518fff4ec88db7f82959da48e05446e44ea2b6dd3ef275e74626acc8f6f6b2c7"}
Jul 16 17:38:09.029241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.028961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"0e7440cee6a5cf7472edada669e263bf86da9b011b8ba1e3ba72a522ba37d2dc"}
Jul 16 17:38:09.029241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.028986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"2b36c572780f1da0269461302c8578588bd24da38bcfb92858b9bcaca5b66313"}
Jul 16 17:38:09.029241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.029010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"0cd752f4d582807463a0163296e745a9aaede9f1d0b8f5183ca2684fb5c77a4d"}
Jul 16 17:38:09.029241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.029036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2a753b5-db7a-4438-9e1b-66e0730df992","Type":"ContainerStarted","Data":"bf6a7da18ea2b79d6eb7ecb780287f8c9ede59a7aa52550416c25bb9831bbf5e"}
Jul 16 17:38:09.067644 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.067602    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.047214571 podStartE2EDuration="2.067586123s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.371552ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:09.065888619 +0000 UTC m=+215.822296683" watchObservedRunningTime="2026-07-16 17:38:09.067586123 +0000 UTC m=+215.823994217"
Jul 16 17:38:09.918853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:09.918818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:10.014079 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:10.014053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:10.586748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:10.586719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:10.665448 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:10.665415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:12.402484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:12.402449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:38:21.248228 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:21.248198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:21.311331 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:21.311312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:21.851009 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:21.850967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:38:47.731807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:38:47.731763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:00.668484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:00.668408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:00.770016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:00.769981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:07.402897 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:07.402855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:39:07.417802 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:07.417781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:39:08.240020 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:08.239994    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 17:39:16.363851 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:16.363811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:16.929677 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:16.929639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:16.985557 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:16.985525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:27.511638 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:27.511598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:27.614516 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:27.614474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:27.677496 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:27.677474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:28.359924 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:28.359890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:28.422997 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:28.422968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:31.663932 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:31.663900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:31.763382 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:31.763351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:31.822775 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:31.822749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:33.213631 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:33.213598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:34.173077 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:34.173052    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 17:39:50.046665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:50.046636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:39:53.404566 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:39:53.404528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:03.111593 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:03.111556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:33.624823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:33.624745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:47.314328 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:47.314290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:47.414204 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:47.414174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:47.657427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:47.657356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:47.746901 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:47.746876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:40:47.848392 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:40:47.848362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:09.571538 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:09.571490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:09.667322 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:09.667291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:15.494839 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:15.494798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:15.824023 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:15.823991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:16.147264 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:16.147179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:16.621585 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:16.621552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.114204 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.114171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.218797 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.218767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.332053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.332023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.472016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.471956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.673024 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.672994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:17.940377 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:17.940343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:18.152547 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:18.152498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:41:18.237397 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:41:18.237323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:27.459200 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:27.459167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:27.560595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:27.560567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:27.971090 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:27.971061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:28.077148 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:28.077119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:28.169394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:28.169357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:46.924143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:46.924107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:42:46.980720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:42:46.980691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:09.842415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:09.842379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:09.952589 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:09.952561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:12.518551 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:12.518516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:12.619707 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:12.619676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:21.026008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:21.025975    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:21.682404 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:21.682372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:23.562159 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:23.562123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:23.668579 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:23.668544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:25.001523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:25.001476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:25.104206 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:25.104171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:26.227208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:26.227168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:26.334966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:26.334935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:26.420762 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:26.420738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:27.196453 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:27.196427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:28.031927 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.031894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:28.359405 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.359336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f8jrq"]
Jul 16 17:43:28.362233 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.362219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.364876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.364853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6kbth\""
Jul 16 17:43:28.365756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.365739    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 17:43:28.365825 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.365783    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 17:43:28.373241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.373217    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f8jrq"]
Jul 16 17:43:28.448053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.448025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.448149 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.448057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9d2\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-kube-api-access-bt9d2\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.548826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.548800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.548826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.548827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9d2\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-kube-api-access-bt9d2\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.566614 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.566585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.566864 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.566848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9d2\" (UniqueName: \"kubernetes.io/projected/aba36d65-4464-4641-bfb6-05be05a3ac3a-kube-api-access-bt9d2\") pod \"cert-manager-webhook-6486ffd997-f8jrq\" (UID: \"aba36d65-4464-4641-bfb6-05be05a3ac3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.670751 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.670688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:28.811313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.811276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f8jrq"]
Jul 16 17:43:28.816175 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:43:28.816149    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba36d65_4464_4641_bfb6_05be05a3ac3a.slice/crio-9ceb42cb46831ed5bb5f00762567a00b8a192b4d26cf1f325e58a5f839db5811 WatchSource:0}: Error finding container 9ceb42cb46831ed5bb5f00762567a00b8a192b4d26cf1f325e58a5f839db5811: Status 404 returned error can't find the container with id 9ceb42cb46831ed5bb5f00762567a00b8a192b4d26cf1f325e58a5f839db5811
Jul 16 17:43:28.818531 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:28.818493    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:43:29.123534 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:29.123488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq" event={"ID":"aba36d65-4464-4641-bfb6-05be05a3ac3a","Type":"ContainerStarted","Data":"9ceb42cb46831ed5bb5f00762567a00b8a192b4d26cf1f325e58a5f839db5811"}
Jul 16 17:43:29.469903 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:29.469823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:29.835691 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:29.835655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:30.043048 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.043016    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"]
Jul 16 17:43:30.048231 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.048207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.055572 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.055548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-94nww\""
Jul 16 17:43:30.061925 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.061900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggn6v\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-kube-api-access-ggn6v\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.062028 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.061969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.070978 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.070957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"]
Jul 16 17:43:30.163256 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.163160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.163693 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.163275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggn6v\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-kube-api-access-ggn6v\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.173973 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.173942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.174108 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.174061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggn6v\" (UniqueName: \"kubernetes.io/projected/958a9f19-e052-446e-b807-cb2832392c98-kube-api-access-ggn6v\") pod \"cert-manager-cainjector-559cdd8c87-ccr24\" (UID: \"958a9f19-e052-446e-b807-cb2832392c98\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:30.360720 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:30.360692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"
Jul 16 17:43:31.741717 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:31.741692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ccr24"]
Jul 16 17:43:31.743762 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:43:31.743731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958a9f19_e052_446e_b807_cb2832392c98.slice/crio-adbe6e508220ca9fe4d086d1a6be885c3039d1f4c6fec98e8b4dbdfa11a2b95c WatchSource:0}: Error finding container adbe6e508220ca9fe4d086d1a6be885c3039d1f4c6fec98e8b4dbdfa11a2b95c: Status 404 returned error can't find the container with id adbe6e508220ca9fe4d086d1a6be885c3039d1f4c6fec98e8b4dbdfa11a2b95c
Jul 16 17:43:31.783185 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:31.783154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:31.883519 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:31.883440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:32.136109 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.136018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq" event={"ID":"aba36d65-4464-4641-bfb6-05be05a3ac3a","Type":"ContainerStarted","Data":"2c0f23b0acebee5e1464fd1d5a576dc05673eab26924ad81d736d8a467b87f1f"}
Jul 16 17:43:32.136276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.136132    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:32.137662 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.137636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24" event={"ID":"958a9f19-e052-446e-b807-cb2832392c98","Type":"ContainerStarted","Data":"93bb140b224ef4fa4274b04669389fb1791508ac49e23b6a771fb084dd448064"}
Jul 16 17:43:32.137753 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.137671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24" event={"ID":"958a9f19-e052-446e-b807-cb2832392c98","Type":"ContainerStarted","Data":"adbe6e508220ca9fe4d086d1a6be885c3039d1f4c6fec98e8b4dbdfa11a2b95c"}
Jul 16 17:43:32.180716 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.180655    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq" podStartSLOduration=1.307770471 podStartE2EDuration="4.180639549s" totalImagesPullingTime="2.872869078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:32.180592738 +0000 UTC m=+538.937000802" watchObservedRunningTime="2026-07-16 17:43:32.180639549 +0000 UTC m=+538.937047613"
Jul 16 17:43:32.226121 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:32.226071    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ccr24" podStartSLOduration=2.226054419 podStartE2EDuration="2.226054419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:32.22588536 +0000 UTC m=+538.982293427" watchObservedRunningTime="2026-07-16 17:43:32.226054419 +0000 UTC m=+538.982462483"
Jul 16 17:43:36.810921 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:36.810884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:36.912238 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:36.912211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:37.631753 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:37.631721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:38.142575 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.142547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f8jrq"
Jul 16 17:43:38.226652 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.226625    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cqhvk"]
Jul 16 17:43:38.229756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.229736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.235902 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.235869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-22n74\""
Jul 16 17:43:38.268167 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.268135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cqhvk"]
Jul 16 17:43:38.324544 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.324523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd84\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-kube-api-access-xqd84\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.324646 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.324557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-bound-sa-token\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.425831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.425763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd84\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-kube-api-access-xqd84\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.425831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.425818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-bound-sa-token\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.440233 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.440210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd84\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-kube-api-access-xqd84\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.440346 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.440296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e85e66e0-a678-4f32-a60d-bdc62611cd80-bound-sa-token\") pod \"cert-manager-55b4d954d5-cqhvk\" (UID: \"e85e66e0-a678-4f32-a60d-bdc62611cd80\") " pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.539047 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.539021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cqhvk"
Jul 16 17:43:38.671112 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:38.671080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cqhvk"]
Jul 16 17:43:38.674636 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:43:38.674611    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode85e66e0_a678_4f32_a60d_bdc62611cd80.slice/crio-5400dcb251e16a7bb85db8c9f410d6a71ccc6c84dca0015ca5b9092a68bfb8a5 WatchSource:0}: Error finding container 5400dcb251e16a7bb85db8c9f410d6a71ccc6c84dca0015ca5b9092a68bfb8a5: Status 404 returned error can't find the container with id 5400dcb251e16a7bb85db8c9f410d6a71ccc6c84dca0015ca5b9092a68bfb8a5
Jul 16 17:43:39.163022 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:39.162991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cqhvk" event={"ID":"e85e66e0-a678-4f32-a60d-bdc62611cd80","Type":"ContainerStarted","Data":"8860d045e71f137c332b3d1d7a1244ec7ef396a310321bc6a73d4542ae101244"}
Jul 16 17:43:39.163362 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:39.163030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cqhvk" event={"ID":"e85e66e0-a678-4f32-a60d-bdc62611cd80","Type":"ContainerStarted","Data":"5400dcb251e16a7bb85db8c9f410d6a71ccc6c84dca0015ca5b9092a68bfb8a5"}
Jul 16 17:43:39.205683 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:39.205632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-cqhvk" podStartSLOduration=1.205616497 podStartE2EDuration="1.205616497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:39.205066332 +0000 UTC m=+545.961474393" watchObservedRunningTime="2026-07-16 17:43:39.205616497 +0000 UTC m=+545.962024560"
Jul 16 17:43:49.836045 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:49.836011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:55.050526 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:55.050472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:55.110807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:55.110776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:56.265954 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:56.265913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:58.881361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:58.881328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:43:58.968912 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:43:58.968879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:02.924353 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:02.924326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:03.008532 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:03.008488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:03.844080 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:03.844049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:03.928319 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:03.928283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:06.451950 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:06.451927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:08.951267 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:08.951230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:09.047293 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:09.047263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:15.143176 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:15.143137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:16.166656 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.166626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:16.234350 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.234322    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"]
Jul 16 17:44:16.234591 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.234570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:16.236605 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.236589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.239078 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.239060    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:44:16.240118 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.240097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 17:44:16.240290 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.240165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 17:44:16.240374 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.240174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 17:44:16.240374 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.240198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-77dwh\""
Jul 16 17:44:16.240374 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.240217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 17:44:16.254791 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.254771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"]
Jul 16 17:44:16.295132 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.295113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rjg\" (UniqueName: \"kubernetes.io/projected/5f574320-981e-484e-ae71-89977f84d9f7-kube-api-access-k7rjg\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.295217 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.295149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.295256 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.295215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.295256 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.295235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f574320-981e-484e-ae71-89977f84d9f7-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.396164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.396135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.396274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.396170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f574320-981e-484e-ae71-89977f84d9f7-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.396274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.396198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rjg\" (UniqueName: \"kubernetes.io/projected/5f574320-981e-484e-ae71-89977f84d9f7-kube-api-access-k7rjg\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.396274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.396240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.396765 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.396744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f574320-981e-484e-ae71-89977f84d9f7-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.398597 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.398578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.398677 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.398609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f574320-981e-484e-ae71-89977f84d9f7-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.430987 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.430935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rjg\" (UniqueName: \"kubernetes.io/projected/5f574320-981e-484e-ae71-89977f84d9f7-kube-api-access-k7rjg\") pod \"lws-controller-manager-5c87b7c6dd-ln2vp\" (UID: \"5f574320-981e-484e-ae71-89977f84d9f7\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.545748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.545717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:16.688917 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:16.688877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"]
Jul 16 17:44:16.691438 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:44:16.691413    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f574320_981e_484e_ae71_89977f84d9f7.slice/crio-e0d07012d484d70dc7652aaad1e57ce3194d710dd357eb1c3af3c4afe780fa5f WatchSource:0}: Error finding container e0d07012d484d70dc7652aaad1e57ce3194d710dd357eb1c3af3c4afe780fa5f: Status 404 returned error can't find the container with id e0d07012d484d70dc7652aaad1e57ce3194d710dd357eb1c3af3c4afe780fa5f
Jul 16 17:44:17.308468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:17.308436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp" event={"ID":"5f574320-981e-484e-ae71-89977f84d9f7","Type":"ContainerStarted","Data":"e0d07012d484d70dc7652aaad1e57ce3194d710dd357eb1c3af3c4afe780fa5f"}
Jul 16 17:44:19.316769 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:19.316735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp" event={"ID":"5f574320-981e-484e-ae71-89977f84d9f7","Type":"ContainerStarted","Data":"728da475a5219c1662bcc16c6b94baa09a5b0d88f8165fc37892207bab035932"}
Jul 16 17:44:19.317112 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:19.316811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:19.353159 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:19.353106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp" podStartSLOduration=0.930988448 podStartE2EDuration="3.353090342s" totalImagesPullingTime="2.422101894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:19.353080111 +0000 UTC m=+586.109488176" watchObservedRunningTime="2026-07-16 17:44:19.353090342 +0000 UTC m=+586.109498405"
Jul 16 17:44:26.203682 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:26.203643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:28.850999 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:28.850965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:29.213478 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:29.213403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:29.630685 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:29.630654    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:30.321935 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:30.321907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-ln2vp"
Jul 16 17:44:32.280939 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:32.280902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:33.010812 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:33.010778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:33.331686 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:33.331611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:33.433682 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:33.433652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:33.539200 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:33.539164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:34.847040 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:34.847003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:36.028249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:36.028219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:36.244574 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:36.244542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:37.785627 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.785591    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"]
Jul 16 17:44:37.791175 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.791156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:37.794184 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.794163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 16 17:44:37.794294 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.794211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 17:44:37.794522 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.794491    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 17:44:37.797057 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.797040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s6h4b\""
Jul 16 17:44:37.813026 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.813002    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"]
Jul 16 17:44:37.972924 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:37.972897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8h6\" (UniqueName: \"kubernetes.io/projected/260439f5-dcdb-498e-b708-760078aa7e35-kube-api-access-6h8h6\") pod \"dns-operator-controller-manager-65b49595d7-7cst2\" (UID: \"260439f5-dcdb-498e-b708-760078aa7e35\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:38.073654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:38.073589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8h6\" (UniqueName: \"kubernetes.io/projected/260439f5-dcdb-498e-b708-760078aa7e35-kube-api-access-6h8h6\") pod \"dns-operator-controller-manager-65b49595d7-7cst2\" (UID: \"260439f5-dcdb-498e-b708-760078aa7e35\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:38.084075 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:38.084046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8h6\" (UniqueName: \"kubernetes.io/projected/260439f5-dcdb-498e-b708-760078aa7e35-kube-api-access-6h8h6\") pod \"dns-operator-controller-manager-65b49595d7-7cst2\" (UID: \"260439f5-dcdb-498e-b708-760078aa7e35\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:38.100612 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:38.100587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:38.233942 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:38.233916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"]
Jul 16 17:44:38.234806 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:44:38.234778    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260439f5_dcdb_498e_b708_760078aa7e35.slice/crio-43af856fe545cf015487c4060c1527ff254f785dffb592b1ac9d10e1aa1de083 WatchSource:0}: Error finding container 43af856fe545cf015487c4060c1527ff254f785dffb592b1ac9d10e1aa1de083: Status 404 returned error can't find the container with id 43af856fe545cf015487c4060c1527ff254f785dffb592b1ac9d10e1aa1de083
Jul 16 17:44:38.383951 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:38.383869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2" event={"ID":"260439f5-dcdb-498e-b708-760078aa7e35","Type":"ContainerStarted","Data":"43af856fe545cf015487c4060c1527ff254f785dffb592b1ac9d10e1aa1de083"}
Jul 16 17:44:39.044885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:39.044851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:39.145343 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:39.145311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:40.121841 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:40.121768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:40.393247 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:40.393168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2" event={"ID":"260439f5-dcdb-498e-b708-760078aa7e35","Type":"ContainerStarted","Data":"9325e86f146cb9c35301ba682cb700005cf5a5d0ede3f9f171b7f39bd35efc7f"}
Jul 16 17:44:40.393247 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:40.393229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:40.414830 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:40.414786    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2" podStartSLOduration=1.794271201 podStartE2EDuration="3.414773498s" totalImagesPullingTime="1.620502297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:40.413311629 +0000 UTC m=+607.169719691" watchObservedRunningTime="2026-07-16 17:44:40.414773498 +0000 UTC m=+607.171181559"
Jul 16 17:44:40.482403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:40.482377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:42.841427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:42.841386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:42.940194 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:42.940167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:43.220997 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:43.220934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:43.319853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:43.319825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:43.427240 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:43.427220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:43.525239 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:43.525210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:44.224537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:44.224488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:44.326065 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:44.326041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:51.399785 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:51.399752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-7cst2"
Jul 16 17:44:51.441135 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:51.441103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:51.526626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:51.526593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:53.872246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:53.872216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:53.931589 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:53.931565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:58.229460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:58.229381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:44:58.332143 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:44:58.332108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:03.630123 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:03.630084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:03.689913 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:03.689883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:04.044234 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:04.044196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:04.140764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:04.140738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:04.429646 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:04.429563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:04.528607 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:04.528575    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:12.476576 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:12.476537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:12.576895 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:12.576869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:14.892252 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:14.892216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:14.992727 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:14.992692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:21.715732 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.715700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:21.738736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.738704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:21.744166 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.744146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.746988 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.746970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-68tkn\""
Jul 16 17:45:21.747100 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.747082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 17:45:21.753283 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.753261    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:21.792215 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.792188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.792327 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.792297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnvw\" (UniqueName: \"kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.830408 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.830381    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:21.842880 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.842856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:21.892780 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.892758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnvw\" (UniqueName: \"kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.892880 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.892822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.893373 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.893357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.902459 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.902432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnvw\" (UniqueName: \"kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw\") pod \"limitador-limitador-ff68ff5cb-ccvfp\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:21.944034 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:21.944011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:22.054146 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.054128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:22.204575 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.204544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:22.211811 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:45:22.211786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69300a08_3313_41f7_abaa_39f2533ac2a2.slice/crio-118b1b16657ff28f8bbac052f2523fcfffe0efbe33d272bf36746e0583421841 WatchSource:0}: Error finding container 118b1b16657ff28f8bbac052f2523fcfffe0efbe33d272bf36746e0583421841: Status 404 returned error can't find the container with id 118b1b16657ff28f8bbac052f2523fcfffe0efbe33d272bf36746e0583421841
Jul 16 17:45:22.547148 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.547109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" event={"ID":"69300a08-3313-41f7-abaa-39f2533ac2a2","Type":"ContainerStarted","Data":"118b1b16657ff28f8bbac052f2523fcfffe0efbe33d272bf36746e0583421841"}
Jul 16 17:45:22.697370 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.697335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:22.798782 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.798680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:22.816187 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.816155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:22.820846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.820831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:22.823223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.823206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4pgm6\""
Jul 16 17:45:22.830619 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.830601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:22.899817 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:22.899789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7j4\" (UniqueName: \"kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4\") pod \"authorino-8df697c9f-rq2g8\" (UID: \"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a\") " pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:23.000684 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:23.000652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7j4\" (UniqueName: \"kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4\") pod \"authorino-8df697c9f-rq2g8\" (UID: \"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a\") " pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:23.011580 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:23.011553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7j4\" (UniqueName: \"kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4\") pod \"authorino-8df697c9f-rq2g8\" (UID: \"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a\") " pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:23.129653 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:23.129594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:23.294626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:23.294590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:23.297289 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:45:23.297245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee44a0d0_03fc_4aaa_9672_641cd2f5ba3a.slice/crio-0679c64b333fc005e871632d39b0ae9341f86e3b1dcf6e6e6fd2d272c17abbd5 WatchSource:0}: Error finding container 0679c64b333fc005e871632d39b0ae9341f86e3b1dcf6e6e6fd2d272c17abbd5: Status 404 returned error can't find the container with id 0679c64b333fc005e871632d39b0ae9341f86e3b1dcf6e6e6fd2d272c17abbd5
Jul 16 17:45:23.552632 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:23.552599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rq2g8" event={"ID":"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a","Type":"ContainerStarted","Data":"0679c64b333fc005e871632d39b0ae9341f86e3b1dcf6e6e6fd2d272c17abbd5"}
Jul 16 17:45:24.831875 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:24.831842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:24.932570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:24.932533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:27.572208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:27.572105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rq2g8" event={"ID":"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a","Type":"ContainerStarted","Data":"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"}
Jul 16 17:45:27.573679 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:27.573654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" event={"ID":"69300a08-3313-41f7-abaa-39f2533ac2a2","Type":"ContainerStarted","Data":"a006f54b5a60c09590643fcf7c3f46d55d6ef2397b030c4769489f78b6723149"}
Jul 16 17:45:27.573784 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:27.573735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:27.589843 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:27.589802    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-rq2g8" podStartSLOduration=1.551555942 podStartE2EDuration="5.589790649s" totalImagesPullingTime="4.038234707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:27.589469058 +0000 UTC m=+654.345877119" watchObservedRunningTime="2026-07-16 17:45:27.589790649 +0000 UTC m=+654.346198711"
Jul 16 17:45:27.610988 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:27.610928    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" podStartSLOduration=1.53584017 podStartE2EDuration="6.610910302s" totalImagesPullingTime="5.075070132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:27.608470808 +0000 UTC m=+654.364878871" watchObservedRunningTime="2026-07-16 17:45:27.610910302 +0000 UTC m=+654.367318366"
Jul 16 17:45:37.814521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:37.814469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:37.815053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:37.814669    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" podUID="69300a08-3313-41f7-abaa-39f2533ac2a2" containerName="limitador" containerID="cri-o://a006f54b5a60c09590643fcf7c3f46d55d6ef2397b030c4769489f78b6723149" gracePeriod=30
Jul 16 17:45:37.815433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:37.815307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:38.617498 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.617457    2577 generic.go:356] "Generic (PLEG): container finished" podID="69300a08-3313-41f7-abaa-39f2533ac2a2" containerID="a006f54b5a60c09590643fcf7c3f46d55d6ef2397b030c4769489f78b6723149" exitCode=0
Jul 16 17:45:38.617718 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.617495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" event={"ID":"69300a08-3313-41f7-abaa-39f2533ac2a2","Type":"ContainerDied","Data":"a006f54b5a60c09590643fcf7c3f46d55d6ef2397b030c4769489f78b6723149"}
Jul 16 17:45:38.744917 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.744896    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:38.828951 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.828928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhnvw\" (UniqueName: \"kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw\") pod \"69300a08-3313-41f7-abaa-39f2533ac2a2\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") "
Jul 16 17:45:38.829248 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.828991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file\") pod \"69300a08-3313-41f7-abaa-39f2533ac2a2\" (UID: \"69300a08-3313-41f7-abaa-39f2533ac2a2\") "
Jul 16 17:45:38.829297 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.829246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file" (OuterVolumeSpecName: "config-file") pod "69300a08-3313-41f7-abaa-39f2533ac2a2" (UID: "69300a08-3313-41f7-abaa-39f2533ac2a2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:45:38.831027 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.831005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw" (OuterVolumeSpecName: "kube-api-access-lhnvw") pod "69300a08-3313-41f7-abaa-39f2533ac2a2" (UID: "69300a08-3313-41f7-abaa-39f2533ac2a2"). InnerVolumeSpecName "kube-api-access-lhnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:45:38.929535 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.929468    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhnvw\" (UniqueName: \"kubernetes.io/projected/69300a08-3313-41f7-abaa-39f2533ac2a2-kube-api-access-lhnvw\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:45:38.929535 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:38.929490    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69300a08-3313-41f7-abaa-39f2533ac2a2-config-file\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:45:39.624881 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:39.624853    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"
Jul 16 17:45:39.625040 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:39.624887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp" event={"ID":"69300a08-3313-41f7-abaa-39f2533ac2a2","Type":"ContainerDied","Data":"118b1b16657ff28f8bbac052f2523fcfffe0efbe33d272bf36746e0583421841"}
Jul 16 17:45:39.625040 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:39.624949    2577 scope.go:117] "RemoveContainer" containerID="a006f54b5a60c09590643fcf7c3f46d55d6ef2397b030c4769489f78b6723149"
Jul 16 17:45:39.648015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:39.647993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:39.653157 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:39.653138    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ccvfp"]
Jul 16 17:45:40.304382 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:40.304349    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69300a08-3313-41f7-abaa-39f2533ac2a2" path="/var/lib/kubelet/pods/69300a08-3313-41f7-abaa-39f2533ac2a2/volumes"
Jul 16 17:45:44.008905 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:44.008870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:45.287850 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:45.287806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:45:48.582729 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.582699    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-zcvdz"]
Jul 16 17:45:48.583084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.582960    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69300a08-3313-41f7-abaa-39f2533ac2a2" containerName="limitador"
Jul 16 17:45:48.583084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.582971    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69300a08-3313-41f7-abaa-39f2533ac2a2" containerName="limitador"
Jul 16 17:45:48.583084 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.583019    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69300a08-3313-41f7-abaa-39f2533ac2a2" containerName="limitador"
Jul 16 17:45:48.586701 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.586684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.589241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.589224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 17:45:48.594842 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.594821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-zcvdz"]
Jul 16 17:45:48.700247 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.700220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87352b8f-8830-4f09-aa00-b144c5a1b06e-tls-cert\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.700390 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.700252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf48l\" (UniqueName: \"kubernetes.io/projected/87352b8f-8830-4f09-aa00-b144c5a1b06e-kube-api-access-kf48l\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.800571 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.800542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87352b8f-8830-4f09-aa00-b144c5a1b06e-tls-cert\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.800695 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.800582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf48l\" (UniqueName: \"kubernetes.io/projected/87352b8f-8830-4f09-aa00-b144c5a1b06e-kube-api-access-kf48l\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.802909 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.802882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87352b8f-8830-4f09-aa00-b144c5a1b06e-tls-cert\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.811120 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.811102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf48l\" (UniqueName: \"kubernetes.io/projected/87352b8f-8830-4f09-aa00-b144c5a1b06e-kube-api-access-kf48l\") pod \"authorino-7c77957575-zcvdz\" (UID: \"87352b8f-8830-4f09-aa00-b144c5a1b06e\") " pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:48.896092 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:48.896040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-zcvdz"
Jul 16 17:45:49.029857 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:49.029820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-zcvdz"]
Jul 16 17:45:49.032635 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:45:49.032605    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87352b8f_8830_4f09_aa00_b144c5a1b06e.slice/crio-3d422598eab8caeb9b8fd73de73700fd370fe0f483e913aec58c18d707e7a65f WatchSource:0}: Error finding container 3d422598eab8caeb9b8fd73de73700fd370fe0f483e913aec58c18d707e7a65f: Status 404 returned error can't find the container with id 3d422598eab8caeb9b8fd73de73700fd370fe0f483e913aec58c18d707e7a65f
Jul 16 17:45:49.663119 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:49.663085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-zcvdz" event={"ID":"87352b8f-8830-4f09-aa00-b144c5a1b06e","Type":"ContainerStarted","Data":"3d422598eab8caeb9b8fd73de73700fd370fe0f483e913aec58c18d707e7a65f"}
Jul 16 17:45:50.668070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:50.668031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-zcvdz" event={"ID":"87352b8f-8830-4f09-aa00-b144c5a1b06e","Type":"ContainerStarted","Data":"fb7303b41d6f07b45d69b4bced6be58df60918a2a222675259e533168bf9d4e1"}
Jul 16 17:45:50.692707 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:50.692664    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-zcvdz" podStartSLOduration=1.902339741 podStartE2EDuration="2.692649827s" totalImagesPullingTime="790.310086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:50.691822867 +0000 UTC m=+677.448230930" watchObservedRunningTime="2026-07-16 17:45:50.692649827 +0000 UTC m=+677.449057888"
Jul 16 17:45:50.722637 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:50.722610    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:50.722785 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:50.722767    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-rq2g8" podUID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" containerName="authorino" containerID="cri-o://7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449" gracePeriod=30
Jul 16 17:45:50.964496 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:50.964472    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:51.119644 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.119615    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg7j4\" (UniqueName: \"kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4\") pod \"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a\" (UID: \"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a\") "
Jul 16 17:45:51.121667 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.121645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4" (OuterVolumeSpecName: "kube-api-access-hg7j4") pod "ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" (UID: "ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a"). InnerVolumeSpecName "kube-api-access-hg7j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:45:51.220412 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.220356    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7j4\" (UniqueName: \"kubernetes.io/projected/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a-kube-api-access-hg7j4\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:45:51.673176 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.673145    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" containerID="7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449" exitCode=0
Jul 16 17:45:51.673536 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.673186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rq2g8" event={"ID":"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a","Type":"ContainerDied","Data":"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"}
Jul 16 17:45:51.673536 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.673199    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rq2g8"
Jul 16 17:45:51.673536 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.673230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rq2g8" event={"ID":"ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a","Type":"ContainerDied","Data":"0679c64b333fc005e871632d39b0ae9341f86e3b1dcf6e6e6fd2d272c17abbd5"}
Jul 16 17:45:51.673536 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.673249    2577 scope.go:117] "RemoveContainer" containerID="7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"
Jul 16 17:45:51.681280 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.681261    2577 scope.go:117] "RemoveContainer" containerID="7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"
Jul 16 17:45:51.681525 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:45:51.681494    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449\": container with ID starting with 7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449 not found: ID does not exist" containerID="7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"
Jul 16 17:45:51.681586 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.681533    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449"} err="failed to get container status \"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449\": rpc error: code = NotFound desc = could not find container \"7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449\": container with ID starting with 7b1aa2eeaf3924bf68c5d7de43150d62b6c6d427233263d772ca8128b55c1449 not found: ID does not exist"
Jul 16 17:45:51.696345 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.696324    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:51.700486 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:51.700467    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rq2g8"]
Jul 16 17:45:52.304260 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:52.304228    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" path="/var/lib/kubelet/pods/ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a/volumes"
Jul 16 17:45:55.705538 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:45:55.705489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:27.909965 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.909887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:27.972166 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.972134    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-29xh8"]
Jul 16 17:46:27.972403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.972392    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" containerName="authorino"
Jul 16 17:46:27.972445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.972405    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" containerName="authorino"
Jul 16 17:46:27.972478 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.972460    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee44a0d0-03fc-4aaa-9672-641cd2f5ba3a" containerName="authorino"
Jul 16 17:46:27.975339 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.975320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:27.975776 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.975756    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w7gqq"]
Jul 16 17:46:27.977754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.977723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fdpb7\""
Jul 16 17:46:27.977754 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.977737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:27.977907 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.977732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 17:46:27.978639 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.978622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:27.978787 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.978772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 16 17:46:27.978838 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.978779    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 17:46:27.980860 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.980844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5pd9n\""
Jul 16 17:46:27.980991 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.980944    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 16 17:46:27.988690 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.988670    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w7gqq"]
Jul 16 17:46:27.991478 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.991457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-29xh8"]
Jul 16 17:46:27.995921 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.995900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqd8f\" (UniqueName: \"kubernetes.io/projected/6c44b3ca-4928-4287-b788-a069ed82cfec-kube-api-access-lqd8f\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:27.996038 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.995967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:27.996038 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.996015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splr7\" (UniqueName: \"kubernetes.io/projected/fe1e8698-c856-4200-8deb-08335f37b1ee-kube-api-access-splr7\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:27.996159 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:27.996068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.097309 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.097219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splr7\" (UniqueName: \"kubernetes.io/projected/fe1e8698-c856-4200-8deb-08335f37b1ee-kube-api-access-splr7\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:28.097481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.097347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.097481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.097387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqd8f\" (UniqueName: \"kubernetes.io/projected/6c44b3ca-4928-4287-b788-a069ed82cfec-kube-api-access-lqd8f\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.097481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.097452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:28.097674 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:46:28.097484    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 16 17:46:28.097674 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:46:28.097567    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 16 17:46:28.097674 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:46:28.097588    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs podName:6c44b3ca-4928-4287-b788-a069ed82cfec nodeName:}" failed. No retries permitted until 2026-07-16 17:46:28.597567247 +0000 UTC m=+715.353975299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs") pod "model-serving-api-fd65d7d6-29xh8" (UID: "6c44b3ca-4928-4287-b788-a069ed82cfec") : secret "model-serving-api-tls" not found
Jul 16 17:46:28.097674 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:46:28.097613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert podName:fe1e8698-c856-4200-8deb-08335f37b1ee nodeName:}" failed. No retries permitted until 2026-07-16 17:46:28.59760132 +0000 UTC m=+715.354009361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert") pod "odh-model-controller-84b54b7f97-w7gqq" (UID: "fe1e8698-c856-4200-8deb-08335f37b1ee") : secret "odh-model-controller-webhook-cert" not found
Jul 16 17:46:28.108698 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.108667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splr7\" (UniqueName: \"kubernetes.io/projected/fe1e8698-c856-4200-8deb-08335f37b1ee-kube-api-access-splr7\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:28.111395 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.111379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqd8f\" (UniqueName: \"kubernetes.io/projected/6c44b3ca-4928-4287-b788-a069ed82cfec-kube-api-access-lqd8f\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.298581 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.298547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:28.408237 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.408210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:28.601713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.601635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:28.601713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.601681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.604015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.603991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe1e8698-c856-4200-8deb-08335f37b1ee-cert\") pod \"odh-model-controller-84b54b7f97-w7gqq\" (UID: \"fe1e8698-c856-4200-8deb-08335f37b1ee\") " pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:28.604130 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.603994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c44b3ca-4928-4287-b788-a069ed82cfec-tls-certs\") pod \"model-serving-api-fd65d7d6-29xh8\" (UID: \"6c44b3ca-4928-4287-b788-a069ed82cfec\") " pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.663883 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.663861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:28.765397 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.765374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:28.886015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.885969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:28.893654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:28.893634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:29.023016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:29.022995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-29xh8"]
Jul 16 17:46:29.025722 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:46:29.025695    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c44b3ca_4928_4287_b788_a069ed82cfec.slice/crio-c5964f3da1fbba7da33704b9d56ef22e1b04a1fb3d6f258185e2c4ba0842bffd WatchSource:0}: Error finding container c5964f3da1fbba7da33704b9d56ef22e1b04a1fb3d6f258185e2c4ba0842bffd: Status 404 returned error can't find the container with id c5964f3da1fbba7da33704b9d56ef22e1b04a1fb3d6f258185e2c4ba0842bffd
Jul 16 17:46:29.039352 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:29.039282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w7gqq"]
Jul 16 17:46:29.041217 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:46:29.041194    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1e8698_c856_4200_8deb_08335f37b1ee.slice/crio-7d32dec80b150d6fc001f840304e2379a56b84e0a5e162d0a0cb466eb45b724d WatchSource:0}: Error finding container 7d32dec80b150d6fc001f840304e2379a56b84e0a5e162d0a0cb466eb45b724d: Status 404 returned error can't find the container with id 7d32dec80b150d6fc001f840304e2379a56b84e0a5e162d0a0cb466eb45b724d
Jul 16 17:46:29.827760 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:29.827638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-29xh8" event={"ID":"6c44b3ca-4928-4287-b788-a069ed82cfec","Type":"ContainerStarted","Data":"c5964f3da1fbba7da33704b9d56ef22e1b04a1fb3d6f258185e2c4ba0842bffd"}
Jul 16 17:46:29.830706 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:29.830675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-w7gqq" event={"ID":"fe1e8698-c856-4200-8deb-08335f37b1ee","Type":"ContainerStarted","Data":"7d32dec80b150d6fc001f840304e2379a56b84e0a5e162d0a0cb466eb45b724d"}
Jul 16 17:46:33.848978 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.848944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-29xh8" event={"ID":"6c44b3ca-4928-4287-b788-a069ed82cfec","Type":"ContainerStarted","Data":"a9d346cd81e2e13445b6c729acba9b7c1f211eddd77f15edf38d93b01ee4651f"}
Jul 16 17:46:33.849401 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.849019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:33.850623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.850594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-w7gqq" event={"ID":"fe1e8698-c856-4200-8deb-08335f37b1ee","Type":"ContainerStarted","Data":"5716b77804652ff79389337df508fcabb98305244e0e916704ce11eb1e95b742"}
Jul 16 17:46:33.850748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.850651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:33.878567 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.878526    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-29xh8" podStartSLOduration=3.034515442 podStartE2EDuration="6.878497528s" totalImagesPullingTime="3.843982086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:33.87721325 +0000 UTC m=+720.633621311" watchObservedRunningTime="2026-07-16 17:46:33.878497528 +0000 UTC m=+720.634905591"
Jul 16 17:46:33.916393 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:33.916357    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-w7gqq" podStartSLOduration=3.139021372 podStartE2EDuration="6.916344973s" totalImagesPullingTime="3.777323601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:33.915824272 +0000 UTC m=+720.672232333" watchObservedRunningTime="2026-07-16 17:46:33.916344973 +0000 UTC m=+720.672753036"
Jul 16 17:46:44.857156 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:44.857125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-w7gqq"
Jul 16 17:46:44.859247 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:44.859224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-29xh8"
Jul 16 17:46:48.691583 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:48.691548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:46:48.790623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:46:48.790576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:03.245113 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:03.245031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:03.346985 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:03.346954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:16.769975 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.769937    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:16.772363 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.772343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.775376 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.775349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-x8k6j\""
Jul 16 17:48:16.775376 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.775369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:48:16.775570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.775391    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:48:16.775570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.775362    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 16 17:48:16.787351 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.787330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:16.813124 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.813098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:16.911367 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.911345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:16.928423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928760 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nv2\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928760 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928760 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.928760 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.928610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:16.984534 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.984493    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:16.986883 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:16.986867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.002569 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.002548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:17.029412 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029412 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029594 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029790 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029790 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nv2\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029790 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029790 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.029985 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.030041 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.029996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.030186 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.030161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.030324 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.030308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.031966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.031948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.032169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.032152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.047060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.047034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nv2\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.048286 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.048268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-xccwl\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.083386 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.083362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:17.130810 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.130810 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvd7\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.130810 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.130985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.131059 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.131021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.217751 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.217727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:17.219082 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:48:17.219058    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933c5a71_7ae4_4aa1_a6f2_e5c1a806daa9.slice/crio-491266d2999492a529379ae454d123221210f701b458ad2b2baf7e687673168d WatchSource:0}: Error finding container 491266d2999492a529379ae454d123221210f701b458ad2b2baf7e687673168d: Status 404 returned error can't find the container with id 491266d2999492a529379ae454d123221210f701b458ad2b2baf7e687673168d
Jul 16 17:48:17.231736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.231816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.231816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.231904 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.231946 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232002 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232002 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.231992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvd7\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232106 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232106 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232208 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232311 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232618 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.232618 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.232615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.234241 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.234222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.234584 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.234568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.243685 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.243667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.243767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.243751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvd7\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7\") pod \"router-gateway-1-openshift-default-54c47cdfc7-9csvf\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.254738 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.254705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" event={"ID":"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9","Type":"ContainerStarted","Data":"491266d2999492a529379ae454d123221210f701b458ad2b2baf7e687673168d"}
Jul 16 17:48:17.296657 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.296608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:17.431682 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:17.431653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:17.435321 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:48:17.435284    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3fa95d_bbd4_479b_9db4_d3402f753c45.slice/crio-2f5aea1679b7f65e2448ed1f5bf217049899635c4aad583b83f7d314a071cbfd WatchSource:0}: Error finding container 2f5aea1679b7f65e2448ed1f5bf217049899635c4aad583b83f7d314a071cbfd: Status 404 returned error can't find the container with id 2f5aea1679b7f65e2448ed1f5bf217049899635c4aad583b83f7d314a071cbfd
Jul 16 17:48:18.263928 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:18.263842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerStarted","Data":"2f5aea1679b7f65e2448ed1f5bf217049899635c4aad583b83f7d314a071cbfd"}
Jul 16 17:48:20.129199 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.129163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:20.179230 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.179197    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.179328 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.179281    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.179328 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.179313    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.185354 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.185332    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.185435 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.185400    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.185472 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.185446    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:20.289297 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.289254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" event={"ID":"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9","Type":"ContainerStarted","Data":"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"}
Jul 16 17:48:20.291015 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.290983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerStarted","Data":"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"}
Jul 16 17:48:20.296686 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.296663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:20.322314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.322254    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" podStartSLOduration=1.3642442049999999 podStartE2EDuration="4.322235583s" totalImagesPullingTime="2.957991378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:20.318457695 +0000 UTC m=+827.074865758" watchObservedRunningTime="2026-07-16 17:48:20.322235583 +0000 UTC m=+827.078643647"
Jul 16 17:48:20.349795 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:20.349717    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" podStartSLOduration=1.601623364 podStartE2EDuration="4.349687025s" totalImagesPullingTime="2.748063661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:20.349136107 +0000 UTC m=+827.105544171" watchObservedRunningTime="2026-07-16 17:48:20.349687025 +0000 UTC m=+827.106095087"
Jul 16 17:48:21.084194 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:21.084156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:21.089536 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:21.089493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:21.294843 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:21.294808    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:21.295490 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:21.295472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:21.302283 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:21.302261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:22.181158 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:22.181124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:22.298314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:22.298282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:22.304136 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:22.304114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:22.378828 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:22.378795    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:24.305521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:24.305476    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" podUID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" containerName="istio-proxy" containerID="cri-o://a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7" gracePeriod=30
Jul 16 17:48:25.754433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:25.754399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:29.604336 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.604283    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:29.655965 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.655927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.655974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656126 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.656394 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.656253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8nv2\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2\") pod \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\" (UID: \"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9\") "
Jul 16 17:48:29.657380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.657278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data" (OuterVolumeSpecName: "istio-data") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:29.657380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.657229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:29.657380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.657351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:29.657665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.657436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:29.657665 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.657541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:29.659876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.659829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:29.659876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.659840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2" (OuterVolumeSpecName: "kube-api-access-t8nv2") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "kube-api-access-t8nv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:29.659993 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.659878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token" (OuterVolumeSpecName: "istio-token") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:29.659993 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.659980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" (UID: "933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 17:48:29.757561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757524    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757561 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757560    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757575    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8nv2\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-kube-api-access-t8nv2\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757590    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757604    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757628    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757640    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757652    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:29.757736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:29.757663    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:30.338971 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.338930    2577 generic.go:356] "Generic (PLEG): container finished" podID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" containerID="a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7" exitCode=0
Jul 16 17:48:30.339118 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.339018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" event={"ID":"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9","Type":"ContainerDied","Data":"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"}
Jul 16 17:48:30.339118 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.339065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl" event={"ID":"933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9","Type":"ContainerDied","Data":"491266d2999492a529379ae454d123221210f701b458ad2b2baf7e687673168d"}
Jul 16 17:48:30.339118 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.339093    2577 scope.go:117] "RemoveContainer" containerID="a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"
Jul 16 17:48:30.339277 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.339259    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"
Jul 16 17:48:30.356584 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.356560    2577 scope.go:117] "RemoveContainer" containerID="a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"
Jul 16 17:48:30.357098 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:48:30.357071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7\": container with ID starting with a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7 not found: ID does not exist" containerID="a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"
Jul 16 17:48:30.357217 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.357109    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7"} err="failed to get container status \"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7\": rpc error: code = NotFound desc = could not find container \"a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7\": container with ID starting with a54352ca6e0d2595af914395bb7671276a674c2f4fd7ef481b181d3cdd678dc7 not found: ID does not exist"
Jul 16 17:48:30.380306 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.380286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:30.381963 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:30.381945    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-xccwl"]
Jul 16 17:48:32.304711 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:32.304684    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" path="/var/lib/kubelet/pods/933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9/volumes"
Jul 16 17:48:35.365181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.365145    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerID="2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9" exitCode=0
Jul 16 17:48:35.365595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.365220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerDied","Data":"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"}
Jul 16 17:48:35.365639 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.365611    2577 scope.go:117] "RemoveContainer" containerID="2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"
Jul 16 17:48:35.366520 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.366479    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:35.366619 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.366571    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:35.366680 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:35.366619    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:36.371327 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:36.371296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerStarted","Data":"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"}
Jul 16 17:48:37.297481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:37.297437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:37.298454 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:37.298430    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:37.298728 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:37.298694    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 16 17:48:37.785864 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:37.785834    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:38.297135 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:38.297102    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 16 17:48:39.297374 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:39.297338    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 16 17:48:39.383476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:39.383433    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy" containerID="cri-o://5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d" gracePeriod=30
Jul 16 17:48:45.755047 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.755010    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:48:45.755459 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.755296    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" containerName="istio-proxy"
Jul 16 17:48:45.755459 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.755308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" containerName="istio-proxy"
Jul 16 17:48:45.755459 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.755375    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="933c5a71-7ae4-4aa1-a6f2-e5c1a806daa9" containerName="istio-proxy"
Jul 16 17:48:45.762694 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.762671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.766016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.765995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:48:45.766140 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.766112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:48:45.766966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.766949    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 16 17:48:45.777572 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.777550    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:48:45.814369 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.814347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:45.900570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9pv\" (UniqueName: \"kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:45.900709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:45.900694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001318 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001440 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001440 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001440 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9pv\" (UniqueName: \"kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001440 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001776 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001839 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001899 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.001953 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.001928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.003644 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.003628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.003778 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.003763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.009874 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.009813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9pv\" (UniqueName: \"kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv\") pod \"precise-prefix-cache-test-kserve-b4979bc47-j2jk4\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.073404 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.073383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:46.170545 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.170514    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:48:46.175138 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.175119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.178474 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.178455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-5x44g\""
Jul 16 17:48:46.187419 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.187398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:48:46.202929 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.202910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:48:46.205383 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:48:46.205359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b4e54c_df57_41e5_bd92_8edf9641d977.slice/crio-9c0f274b84d5709c37efb8e4e7c7383381915f9f3868a3b6f8fa9c8a4277a0db WatchSource:0}: Error finding container 9c0f274b84d5709c37efb8e4e7c7383381915f9f3868a3b6f8fa9c8a4277a0db: Status 404 returned error can't find the container with id 9c0f274b84d5709c37efb8e4e7c7383381915f9f3868a3b6f8fa9c8a4277a0db
Jul 16 17:48:46.207246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.207232    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:48:46.296236 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.296180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:46.303966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.303945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.304070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.303982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.304070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.304010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.304070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.304057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.304233 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.304108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqnv\" (UniqueName: \"kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.304233 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.304180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404688 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404800 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404800 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404800 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404941 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqnv\" (UniqueName: \"kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.404988 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.404955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.405097 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.405065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.405185 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.405106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.405238 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.405210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.405335 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.405315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.407403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.407381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.409747 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.409727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerStarted","Data":"9c0f274b84d5709c37efb8e4e7c7383381915f9f3868a3b6f8fa9c8a4277a0db"}
Jul 16 17:48:46.414805 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.414778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqnv\" (UniqueName: \"kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.484931 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.484911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:48:46.618627 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.618591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:48:46.619729 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:48:46.619704    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3685774b_11b2_49d5_9395_84ad14bd1823.slice/crio-a9d112747a43be2b2a2aa0699cc3083f6f166405ad909ff733c04e6b950e1b3f WatchSource:0}: Error finding container a9d112747a43be2b2a2aa0699cc3083f6f166405ad909ff733c04e6b950e1b3f: Status 404 returned error can't find the container with id a9d112747a43be2b2a2aa0699cc3083f6f166405ad909ff733c04e6b950e1b3f
Jul 16 17:48:46.732674 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.732645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:46.808426 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808473    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkvd7\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo\") pod \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\" (UID: \"5d3fa95d-bbd4-479b-9db4-d3402f753c45\") "
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:46.808823 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:46.809365 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:46.809365 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.808992    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.809365 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.809010    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.809365 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.809023    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.809365 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.809052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:46.810661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.810640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:46.810765 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.810676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7" (OuterVolumeSpecName: "kube-api-access-bkvd7") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "kube-api-access-bkvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:46.810765 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.810727    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 17:48:46.810854 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.810803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d3fa95d-bbd4-479b-9db4-d3402f753c45" (UID: "5d3fa95d-bbd4-479b-9db4-d3402f753c45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909780    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909801    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvd7\" (UniqueName: \"kubernetes.io/projected/5d3fa95d-bbd4-479b-9db4-d3402f753c45-kube-api-access-bkvd7\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909811    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909819    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909827    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:46.909833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:46.909834    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3fa95d-bbd4-479b-9db4-d3402f753c45-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:48:47.076100 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.076003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:47.418623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.418424    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerID="5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d" exitCode=0
Jul 16 17:48:47.418623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.418608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerDied","Data":"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"}
Jul 16 17:48:47.418871 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.418666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf" event={"ID":"5d3fa95d-bbd4-479b-9db4-d3402f753c45","Type":"ContainerDied","Data":"2f5aea1679b7f65e2448ed1f5bf217049899635c4aad583b83f7d314a071cbfd"}
Jul 16 17:48:47.418871 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.418698    2577 scope.go:117] "RemoveContainer" containerID="5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"
Jul 16 17:48:47.418871 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.418816    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"
Jul 16 17:48:47.421788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.421709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"a9d112747a43be2b2a2aa0699cc3083f6f166405ad909ff733c04e6b950e1b3f"}
Jul 16 17:48:47.433417 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.433395    2577 scope.go:117] "RemoveContainer" containerID="2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"
Jul 16 17:48:47.451190 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.451143    2577 scope.go:117] "RemoveContainer" containerID="5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"
Jul 16 17:48:47.452008 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:48:47.451981    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d\": container with ID starting with 5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d not found: ID does not exist" containerID="5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"
Jul 16 17:48:47.452111 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.452021    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d"} err="failed to get container status \"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d\": rpc error: code = NotFound desc = could not find container \"5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d\": container with ID starting with 5fd3aa020416731ada9594e7104618786c277e2830ed3d40ad9d72f80f1ff37d not found: ID does not exist"
Jul 16 17:48:47.452111 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.452045    2577 scope.go:117] "RemoveContainer" containerID="2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"
Jul 16 17:48:47.454124 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:48:47.454071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9\": container with ID starting with 2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9 not found: ID does not exist" containerID="2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"
Jul 16 17:48:47.454124 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.454101    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9"} err="failed to get container status \"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9\": rpc error: code = NotFound desc = could not find container \"2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9\": container with ID starting with 2f2e1a495cf443257fbb0dfbb3bdf206088e6f4242a03824cd63332eb0d1b5f9 not found: ID does not exist"
Jul 16 17:48:47.454930 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.454888    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:47.457052 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:47.457035    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-9csvf"]
Jul 16 17:48:48.307267 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:48.307128    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" path="/var/lib/kubelet/pods/5d3fa95d-bbd4-479b-9db4-d3402f753c45/volumes"
Jul 16 17:48:48.957874 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:48.957835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:50.444248 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:50.444210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerStarted","Data":"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"}
Jul 16 17:48:50.445959 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:50.445931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"}
Jul 16 17:48:51.454415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:51.454148    2577 generic.go:356] "Generic (PLEG): container finished" podID="3685774b-11b2-49d5-9395-84ad14bd1823" containerID="1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5" exitCode=0
Jul 16 17:48:51.454415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:51.454236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"}
Jul 16 17:48:52.598923 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:52.598810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:54.468605 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:54.468526    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerID="16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e" exitCode=0
Jul 16 17:48:54.468605 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:54.468589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerDied","Data":"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"}
Jul 16 17:48:54.470431 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:54.470413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"82b4b882b6cb3b852d720fced8782a41aa967efbba90b16132cd3051b945a139"}
Jul 16 17:48:55.833200 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833162    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 17:48:55.833661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.833661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833601    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.833661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.833661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833635    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.834108 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833710    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.834108 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.833720    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3fa95d-bbd4-479b-9db4-d3402f753c45" containerName="istio-proxy"
Jul 16 17:48:55.838368 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.838345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.841537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.841480    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 16 17:48:55.842000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.841815    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-x8k6j\""
Jul 16 17:48:55.863824 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.863759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 17:48:55.997789 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.997760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.997966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.997801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvx8\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.997966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.997825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.997966 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.997939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.998127 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.997978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.998127 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.998003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.998127 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.998042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.998127 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.998088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:55.998127 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:55.998110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvx8\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.099756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.100060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.099886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.100283 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.100257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.100418 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.100300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.100418 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.100332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.100752 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.100723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.102042 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.102020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.102380 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.102357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.110079 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.110053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvx8\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.112451 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.112429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-p4vzh\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.152403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.152379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:56.929188 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.929160    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 17:48:56.931331 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:48:56.931302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123399d5_de98_4560_b7e3_37814a765e9b.slice/crio-6d85d8b42b8829907697611bcf568061284a265a1d98c9754e4232bae0129316 WatchSource:0}: Error finding container 6d85d8b42b8829907697611bcf568061284a265a1d98c9754e4232bae0129316: Status 404 returned error can't find the container with id 6d85d8b42b8829907697611bcf568061284a265a1d98c9754e4232bae0129316
Jul 16 17:48:56.933933 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.933851    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:56.934037 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.933933    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:56.934037 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:56.933981    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:48:57.485934 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:57.485893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerStarted","Data":"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"}
Jul 16 17:48:57.486118 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:57.485961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerStarted","Data":"6d85d8b42b8829907697611bcf568061284a265a1d98c9754e4232bae0129316"}
Jul 16 17:48:57.487574 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:57.487550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerStarted","Data":"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"}
Jul 16 17:48:57.513844 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:57.513771    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podStartSLOduration=2.51375939 podStartE2EDuration="2.51375939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:57.511119136 +0000 UTC m=+864.267527211" watchObservedRunningTime="2026-07-16 17:48:57.51375939 +0000 UTC m=+864.270167451"
Jul 16 17:48:57.533774 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:57.533737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" podStartSLOduration=2.931086186 podStartE2EDuration="12.533724824s" totalImagesPullingTime="6.338687008s" totalInitContainerRuntime="3.26395163s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:57.533528874 +0000 UTC m=+864.289936933" watchObservedRunningTime="2026-07-16 17:48:57.533724824 +0000 UTC m=+864.290132886"
Jul 16 17:48:58.074383 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.074355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:48:58.152822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.152786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:58.159283 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.159257    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:58.491704 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.491620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:58.492573 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.492549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:48:58.965773 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:58.965733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:48:59.038560 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:48:59.038530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:02.348967 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:02.348925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:04.010446 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:04.010413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:04.522842 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:04.522814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:04.523165 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:04.523143    2577 generic.go:356] "Generic (PLEG): container finished" podID="3685774b-11b2-49d5-9395-84ad14bd1823" containerID="82b4b882b6cb3b852d720fced8782a41aa967efbba90b16132cd3051b945a139" exitCode=1
Jul 16 17:49:04.523235 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:04.523214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"82b4b882b6cb3b852d720fced8782a41aa967efbba90b16132cd3051b945a139"}
Jul 16 17:49:06.073578 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:06.073534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:49:06.093257 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:06.093234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:49:06.546382 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:06.546354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:49:17.648564 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.648527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:17.899831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.899757    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:49:17.909074 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.909054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:17.911773 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.911748    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 16 17:49:17.912737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.912717    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-4xc49\""
Jul 16 17:49:17.918768 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.918747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:49:17.947796 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.947770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:17.990581 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.990553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:17.990738 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.990614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:17.990738 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.990718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:17.990849 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:17.990753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v654f\" (UniqueName: \"kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092216 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092216 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v654f\" (UniqueName: \"kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.092785 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.092681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.095065 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.095040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.104056 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.104029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v654f\" (UniqueName: \"kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f\") pod \"auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:18.220274 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:18.220184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:25.531484 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:25.531402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:32.017415 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.017391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:49:32.026689 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:49:32.026665    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e33848_a5b5_440d_aa29_3b8fa28a5339.slice/crio-6fee5b17e43ccf1df8db8ac88304bc0d45b3e0d4430b2b52cfdebe0c5c0e94d5 WatchSource:0}: Error finding container 6fee5b17e43ccf1df8db8ac88304bc0d45b3e0d4430b2b52cfdebe0c5c0e94d5: Status 404 returned error can't find the container with id 6fee5b17e43ccf1df8db8ac88304bc0d45b3e0d4430b2b52cfdebe0c5c0e94d5
Jul 16 17:49:32.663433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.663404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:32.663895 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.663869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"}
Jul 16 17:49:32.664215 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.664193    2577 scope.go:117] "RemoveContainer" containerID="82b4b882b6cb3b852d720fced8782a41aa967efbba90b16132cd3051b945a139"
Jul 16 17:49:32.665630 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.665607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerStarted","Data":"15d8b09ebed29d8cbf383c4d51019b73374f8901b71c0d9f00536d0f51877b1a"}
Jul 16 17:49:32.665719 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:32.665651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerStarted","Data":"6fee5b17e43ccf1df8db8ac88304bc0d45b3e0d4430b2b52cfdebe0c5c0e94d5"}
Jul 16 17:49:33.673933 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.673900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:33.674368 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.674274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"}
Jul 16 17:49:33.675164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.675144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:33.677132 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.677102    2577 generic.go:356] "Generic (PLEG): container finished" podID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerID="15d8b09ebed29d8cbf383c4d51019b73374f8901b71c0d9f00536d0f51877b1a" exitCode=0
Jul 16 17:49:33.677231 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.677213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerDied","Data":"15d8b09ebed29d8cbf383c4d51019b73374f8901b71c0d9f00536d0f51877b1a"}
Jul 16 17:49:33.724701 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.724649    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podStartSLOduration=2.884364805 podStartE2EDuration="47.724634077s" totalImagesPullingTime="44.058749156s" totalInitContainerRuntime="781.520116ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:33.702328372 +0000 UTC m=+900.458736435" watchObservedRunningTime="2026-07-16 17:49:33.724634077 +0000 UTC m=+900.481042138"
Jul 16 17:49:33.954571 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:33.954476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:49:34.220436 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:34.220360    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:34.220822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:34.220800    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:34.684152 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:34.684117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerStarted","Data":"ebc815df88864e0d057be0f67e09ccf02a942e9aa3ac8bdb370c53163800fcc3"}
Jul 16 17:49:34.684527 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:34.684301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:49:34.705181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:34.705138    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" podStartSLOduration=16.968451495 podStartE2EDuration="17.705124035s" totalImagesPullingTime="0s" totalInitContainerRuntime="736.67254ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:34.703865586 +0000 UTC m=+901.460273669" watchObservedRunningTime="2026-07-16 17:49:34.705124035 +0000 UTC m=+901.461532097"
Jul 16 17:49:36.485321 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:36.485282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:36.485713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:36.485374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:43.726557 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.726531    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:49:43.727235 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.727220    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/0.log"
Jul 16 17:49:43.727571 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.727549    2577 generic.go:356] "Generic (PLEG): container finished" podID="3685774b-11b2-49d5-9395-84ad14bd1823" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd" exitCode=1
Jul 16 17:49:43.727640 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.727618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"}
Jul 16 17:49:43.727690 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.727680    2577 scope.go:117] "RemoveContainer" containerID="82b4b882b6cb3b852d720fced8782a41aa967efbba90b16132cd3051b945a139"
Jul 16 17:49:43.728070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:43.728050    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:49:43.728286 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:49:43.728263    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_kserve-ci-e2e-test(3685774b-11b2-49d5-9395-84ad14bd1823)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823"
Jul 16 17:49:44.733820 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:44.733787    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:49:46.485916 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:46.485883    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:46.486366 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:46.486349    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:49:46.486621 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:49:46.486600    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_kserve-ci-e2e-test(3685774b-11b2-49d5-9395-84ad14bd1823)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823"
Jul 16 17:49:46.488625 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:46.488602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:46.743791 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:46.743722    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:49:46.743944 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:49:46.743899    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_kserve-ci-e2e-test(3685774b-11b2-49d5-9395-84ad14bd1823)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823"
Jul 16 17:49:46.744706 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:46.744686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:49:47.747410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:49:47.747385    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:49:47.747782 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:49:47.747602    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_kserve-ci-e2e-test(3685774b-11b2-49d5-9395-84ad14bd1823)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823"
Jul 16 17:50:01.300314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:01.300279    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:50:01.816767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:01.816741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:50:01.817377 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:01.817350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerStarted","Data":"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"}
Jul 16 17:50:01.817669 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:01.817652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:50:05.690662 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:05.690635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:50:06.487237 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:50:06.487206    2577 logging.go:55] [core] [Channel #29 SubChannel #30]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.28:9003", ServerName: "10.132.0.28:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.28:9003: connect: connection refused"
Jul 16 17:50:07.486032 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:07.485975    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.28:9003\" within 1s: context deadline exceeded"
Jul 16 17:50:32.823678 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:32.823647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:50:34.455627 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:34.455584    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:50:34.456458 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:34.456423    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer" containerID="cri-o://110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6" gracePeriod=60
Jul 16 17:50:34.458388 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:34.458359    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:50:34.469991 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:34.469962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:50:34.570454 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:34.570420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:50:35.968815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:35.968787    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:50:35.969432 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:35.969411    2577 generic.go:356] "Generic (PLEG): container finished" podID="3685774b-11b2-49d5-9395-84ad14bd1823" containerID="110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6" exitCode=0
Jul 16 17:50:35.969487 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:35.969450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"}
Jul 16 17:50:36.743881 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:36.743834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.28:8082/healthz\": dial tcp 10.132.0.28:8082: connect: connection refused"
Jul 16 17:50:42.823304 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:42.823266    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:50:46.744203 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:46.744160    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.28:8082/healthz\": dial tcp 10.132.0.28:8082: connect: connection refused"
Jul 16 17:50:49.481650 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.481611    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="main" containerID="cri-o://677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86" gracePeriod=45
Jul 16 17:50:49.483900 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.483855    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main" containerID="cri-o://b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c" gracePeriod=45
Jul 16 17:50:49.533570 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.533540    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="main" probeResult="failure" output="Get \"https://10.132.0.27:8000/health\": dial tcp 10.132.0.27:8000: connect: connection refused"
Jul 16 17:50:49.753753 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.753732    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:50:49.756003 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.755980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:50:49.758045 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.758028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:50:49.877727 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.877853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.877853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.877853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.877853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877958    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.877983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clqnv\" (UniqueName: \"kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878062 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq9pv\" (UniqueName: \"kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv\") pod \"e6b4e54c-df57-41e5-bd92-8edf9641d977\" (UID: \"e6b4e54c-df57-41e5-bd92-8edf9641d977\") "
Jul 16 17:50:49.878361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp\") pod \"3685774b-11b2-49d5-9395-84ad14bd1823\" (UID: \"3685774b-11b2-49d5-9395-84ad14bd1823\") "
Jul 16 17:50:49.878361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.878361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.878361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.878660 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878406    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.878660 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878425    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.878660 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878440    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.878808 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878758    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.878940 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home" (OuterVolumeSpecName: "home") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.879008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.878952    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.879072 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.879042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.880250 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.880227    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:50:49.880436 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.880419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm" (OuterVolumeSpecName: "dshm") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.880625 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.880608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:50:49.880690 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.880661    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv" (OuterVolumeSpecName: "kube-api-access-clqnv") pod "3685774b-11b2-49d5-9395-84ad14bd1823" (UID: "3685774b-11b2-49d5-9395-84ad14bd1823"). InnerVolumeSpecName "kube-api-access-clqnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:50:49.880988 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.880971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv" (OuterVolumeSpecName: "kube-api-access-cq9pv") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "kube-api-access-cq9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:50:49.934017 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.933986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6b4e54c-df57-41e5-bd92-8edf9641d977" (UID: "e6b4e54c-df57-41e5-bd92-8edf9641d977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.979224 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979197    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979224 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979223    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3685774b-11b2-49d5-9395-84ad14bd1823-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979235    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqnv\" (UniqueName: \"kubernetes.io/projected/3685774b-11b2-49d5-9395-84ad14bd1823-kube-api-access-clqnv\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979243    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979254    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9pv\" (UniqueName: \"kubernetes.io/projected/e6b4e54c-df57-41e5-bd92-8edf9641d977-kube-api-access-cq9pv\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979262    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3685774b-11b2-49d5-9395-84ad14bd1823-tokenizer-tmp\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979270    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b4e54c-df57-41e5-bd92-8edf9641d977-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979278    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979285    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.979364 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:49.979292    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b4e54c-df57-41e5-bd92-8edf9641d977-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:50:50.030618 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.030591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7_3685774b-11b2-49d5-9395-84ad14bd1823/main/1.log"
Jul 16 17:50:50.031248 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.031224    2577 generic.go:356] "Generic (PLEG): container finished" podID="3685774b-11b2-49d5-9395-84ad14bd1823" containerID="b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c" exitCode=0
Jul 16 17:50:50.031357 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.031285    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"
Jul 16 17:50:50.031357 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.031309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"}
Jul 16 17:50:50.031448 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.031373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7" event={"ID":"3685774b-11b2-49d5-9395-84ad14bd1823","Type":"ContainerDied","Data":"a9d112747a43be2b2a2aa0699cc3083f6f166405ad909ff733c04e6b950e1b3f"}
Jul 16 17:50:50.031448 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.031407    2577 scope.go:117] "RemoveContainer" containerID="b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"
Jul 16 17:50:50.032922 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.032899    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerID="677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86" exitCode=0
Jul 16 17:50:50.033025 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.032954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"
Jul 16 17:50:50.033025 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.032964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerDied","Data":"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"}
Jul 16 17:50:50.033025 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.032997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4" event={"ID":"e6b4e54c-df57-41e5-bd92-8edf9641d977","Type":"ContainerDied","Data":"9c0f274b84d5709c37efb8e4e7c7383381915f9f3868a3b6f8fa9c8a4277a0db"}
Jul 16 17:50:50.042018 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.042004    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:50:50.049844 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.049825    2577 scope.go:117] "RemoveContainer" containerID="110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"
Jul 16 17:50:50.056422 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.056407    2577 scope.go:117] "RemoveContainer" containerID="1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"
Jul 16 17:50:50.063428 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.063410    2577 scope.go:117] "RemoveContainer" containerID="b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"
Jul 16 17:50:50.063709 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.063687    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c\": container with ID starting with b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c not found: ID does not exist" containerID="b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"
Jul 16 17:50:50.063783 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.063717    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c"} err="failed to get container status \"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c\": rpc error: code = NotFound desc = could not find container \"b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c\": container with ID starting with b0860ec0f31581a920276ee3b19d44d77f4279050b679b5fd656c4dff10d354c not found: ID does not exist"
Jul 16 17:50:50.063783 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.063734    2577 scope.go:117] "RemoveContainer" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:50:50.063971 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.063958    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd\": container with ID starting with 9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd not found: ID does not exist" containerID="9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"
Jul 16 17:50:50.064013 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.063975    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd"} err="failed to get container status \"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd\": rpc error: code = NotFound desc = could not find container \"9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd\": container with ID starting with 9d2ca3a822807c28913656f92307a3ec35ed4d6cd9162ece0543667bee8dc9bd not found: ID does not exist"
Jul 16 17:50:50.064013 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.063988    2577 scope.go:117] "RemoveContainer" containerID="110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"
Jul 16 17:50:50.064197 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.064184    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6\": container with ID starting with 110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6 not found: ID does not exist" containerID="110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"
Jul 16 17:50:50.064276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.064199    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6"} err="failed to get container status \"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6\": rpc error: code = NotFound desc = could not find container \"110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6\": container with ID starting with 110d13db1cbfb14bd7b2b1d7053d400f503faf691f2686a4380f5299c0bd06f6 not found: ID does not exist"
Jul 16 17:50:50.064276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.064210    2577 scope.go:117] "RemoveContainer" containerID="1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"
Jul 16 17:50:50.064424 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.064402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5\": container with ID starting with 1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5 not found: ID does not exist" containerID="1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"
Jul 16 17:50:50.064463 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.064428    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5"} err="failed to get container status \"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5\": rpc error: code = NotFound desc = could not find container \"1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5\": container with ID starting with 1a762bdc24e78342d213daa964dd178da580d646f053b9d667f5ee8251d403b5 not found: ID does not exist"
Jul 16 17:50:50.064463 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.064439    2577 scope.go:117] "RemoveContainer" containerID="677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"
Jul 16 17:50:50.065295 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.065278    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:50:50.071180 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.071165    2577 scope.go:117] "RemoveContainer" containerID="16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"
Jul 16 17:50:50.078249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.078227    2577 scope.go:117] "RemoveContainer" containerID="677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"
Jul 16 17:50:50.078532 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.078483    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86\": container with ID starting with 677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86 not found: ID does not exist" containerID="677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"
Jul 16 17:50:50.078603 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.078530    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86"} err="failed to get container status \"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86\": rpc error: code = NotFound desc = could not find container \"677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86\": container with ID starting with 677fa290da6e5d26ef1036611f7c326bdc033da9c5fb0e9cc2a6e28c3f983b86 not found: ID does not exist"
Jul 16 17:50:50.078603 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.078547    2577 scope.go:117] "RemoveContainer" containerID="16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"
Jul 16 17:50:50.078788 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:50:50.078762    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e\": container with ID starting with 16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e not found: ID does not exist" containerID="16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"
Jul 16 17:50:50.078838 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.078798    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e"} err="failed to get container status \"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e\": rpc error: code = NotFound desc = could not find container \"16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e\": container with ID starting with 16c43f6d423237d919d867429b7558f023c3575a7463a2f510e7dce18262f13e not found: ID does not exist"
Jul 16 17:50:50.079255 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.079240    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-j2jk4"]
Jul 16 17:50:50.101474 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.101452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:50:50.117392 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.117371    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b4b884cw59s7"]
Jul 16 17:50:50.303944 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.303874    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" path="/var/lib/kubelet/pods/3685774b-11b2-49d5-9395-84ad14bd1823/volumes"
Jul 16 17:50:50.304344 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:50:50.304332    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" path="/var/lib/kubelet/pods/e6b4e54c-df57-41e5-bd92-8edf9641d977/volumes"
Jul 16 17:51:03.023306 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.023232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:03.350040 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.349962    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:03.350246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350236    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350248    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350259    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350264    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350270    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="main"
Jul 16 17:51:03.350296 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350290    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="main"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350298    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="storage-initializer"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350305    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="storage-initializer"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350312    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="storage-initializer"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350317    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="storage-initializer"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350380    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b4e54c-df57-41e5-bd92-8edf9641d977" containerName="main"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:51:03.350493 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.350407    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="tokenizer"
Jul 16 17:51:03.352358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.352343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.356067 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.356046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 16 17:51:03.356067 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.356058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-f9tfp\""
Jul 16 17:51:03.363135 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.363112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:03.375931 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.375907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.376031 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.375960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.376031 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.375980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrbp\" (UniqueName: \"kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.476591 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.476541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.476771 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.476646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.476771 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.476674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrbp\" (UniqueName: \"kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.477038 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.477018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.478845 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.478824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.487098 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.487069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrbp\" (UniqueName: \"kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.661846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.661761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:03.789574 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:03.789544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:03.792486 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:51:03.792462    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3117f868_22c8_46e2_97a5_b2966c47e40d.slice/crio-28451fc07a1d31be894f1aa947aa89ec090400fa7247e6013ce22211f7327ccb WatchSource:0}: Error finding container 28451fc07a1d31be894f1aa947aa89ec090400fa7247e6013ce22211f7327ccb: Status 404 returned error can't find the container with id 28451fc07a1d31be894f1aa947aa89ec090400fa7247e6013ce22211f7327ccb
Jul 16 17:51:04.096825 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:04.096791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" event={"ID":"3117f868-22c8-46e2-97a5-b2966c47e40d","Type":"ContainerStarted","Data":"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"}
Jul 16 17:51:04.097297 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:04.096845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" event={"ID":"3117f868-22c8-46e2-97a5-b2966c47e40d","Type":"ContainerStarted","Data":"28451fc07a1d31be894f1aa947aa89ec090400fa7247e6013ce22211f7327ccb"}
Jul 16 17:51:04.097297 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:04.096918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:04.121444 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:04.121377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" podStartSLOduration=1.121357535 podStartE2EDuration="1.121357535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:51:04.120317925 +0000 UTC m=+990.876725991" watchObservedRunningTime="2026-07-16 17:51:04.121357535 +0000 UTC m=+990.877765598"
Jul 16 17:51:04.227117 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:04.227084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:11.493456 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:11.493423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:12.566849 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:12.566816    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:35.102602 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:35.102575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:36.940688 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:36.940656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:36.941123 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:36.940668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:37.002938 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:37.002902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:43.125322 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:43.125287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:43.166943 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:43.166910    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:51:43.211737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:43.211706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:45.102902 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:45.102860    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:51:45.690627 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:45.690595    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:51:51.960283 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:51.960241    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerName="main" containerID="cri-o://8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d" gracePeriod=45
Jul 16 17:51:52.215065 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.215004    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:52.303995 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.303957    2577 generic.go:356] "Generic (PLEG): container finished" podID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerID="8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d" exitCode=0
Jul 16 17:51:52.304182 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.304067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"
Jul 16 17:51:52.305950 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.305930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" event={"ID":"3117f868-22c8-46e2-97a5-b2966c47e40d","Type":"ContainerDied","Data":"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"}
Jul 16 17:51:52.306031 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.305972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v" event={"ID":"3117f868-22c8-46e2-97a5-b2966c47e40d","Type":"ContainerDied","Data":"28451fc07a1d31be894f1aa947aa89ec090400fa7247e6013ce22211f7327ccb"}
Jul 16 17:51:52.306031 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.305992    2577 scope.go:117] "RemoveContainer" containerID="8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"
Jul 16 17:51:52.313716 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.313700    2577 scope.go:117] "RemoveContainer" containerID="8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"
Jul 16 17:51:52.313954 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:51:52.313935    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d\": container with ID starting with 8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d not found: ID does not exist" containerID="8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"
Jul 16 17:51:52.314023 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.313967    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d"} err="failed to get container status \"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d\": rpc error: code = NotFound desc = could not find container \"8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d\": container with ID starting with 8ee09fe2ec29fe5f893081a5650007dbad465e13810574d5219fe816edac434d not found: ID does not exist"
Jul 16 17:51:52.367331 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.367307    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds\") pod \"3117f868-22c8-46e2-97a5-b2966c47e40d\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") "
Jul 16 17:51:52.367466 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.367343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqrbp\" (UniqueName: \"kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp\") pod \"3117f868-22c8-46e2-97a5-b2966c47e40d\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") "
Jul 16 17:51:52.367466 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.367374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs\") pod \"3117f868-22c8-46e2-97a5-b2966c47e40d\" (UID: \"3117f868-22c8-46e2-97a5-b2966c47e40d\") "
Jul 16 17:51:52.367649 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.367626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3117f868-22c8-46e2-97a5-b2966c47e40d" (UID: "3117f868-22c8-46e2-97a5-b2966c47e40d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:52.369387 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.369354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp" (OuterVolumeSpecName: "kube-api-access-pqrbp") pod "3117f868-22c8-46e2-97a5-b2966c47e40d" (UID: "3117f868-22c8-46e2-97a5-b2966c47e40d"). InnerVolumeSpecName "kube-api-access-pqrbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:51:52.369387 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.369363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3117f868-22c8-46e2-97a5-b2966c47e40d" (UID: "3117f868-22c8-46e2-97a5-b2966c47e40d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:51:52.468246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.468160    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3117f868-22c8-46e2-97a5-b2966c47e40d-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:52.468246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.468190    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrbp\" (UniqueName: \"kubernetes.io/projected/3117f868-22c8-46e2-97a5-b2966c47e40d-kube-api-access-pqrbp\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:52.468246 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.468199    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3117f868-22c8-46e2-97a5-b2966c47e40d-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:52.627876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.627842    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:52.632919 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:52.632895    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehdj9v"]
Jul 16 17:51:54.305216 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:54.305172    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" path="/var/lib/kubelet/pods/3117f868-22c8-46e2-97a5-b2966c47e40d/volumes"
Jul 16 17:51:54.947654 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:54.947622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:55.053071 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:55.053044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:51:55.690124 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:55.690087    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:51:58.186368 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.186325    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main" containerID="cri-o://ebc815df88864e0d057be0f67e09ccf02a942e9aa3ac8bdb370c53163800fcc3" gracePeriod=45
Jul 16 17:51:58.331982 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.331951    2577 generic.go:356] "Generic (PLEG): container finished" podID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerID="ebc815df88864e0d057be0f67e09ccf02a942e9aa3ac8bdb370c53163800fcc3" exitCode=0
Jul 16 17:51:58.332130 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.332030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerDied","Data":"ebc815df88864e0d057be0f67e09ccf02a942e9aa3ac8bdb370c53163800fcc3"}
Jul 16 17:51:58.427645 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.427623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:51:58.510390 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.510364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs\") pod \"81e33848-a5b5-440d-aa29-3b8fa28a5339\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") "
Jul 16 17:51:58.510532 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.510421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location\") pod \"81e33848-a5b5-440d-aa29-3b8fa28a5339\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") "
Jul 16 17:51:58.510532 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.510484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds\") pod \"81e33848-a5b5-440d-aa29-3b8fa28a5339\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") "
Jul 16 17:51:58.510655 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.510529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v654f\" (UniqueName: \"kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f\") pod \"81e33848-a5b5-440d-aa29-3b8fa28a5339\" (UID: \"81e33848-a5b5-440d-aa29-3b8fa28a5339\") "
Jul 16 17:51:58.510807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.510772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "81e33848-a5b5-440d-aa29-3b8fa28a5339" (UID: "81e33848-a5b5-440d-aa29-3b8fa28a5339"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:58.511178 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.511155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81e33848-a5b5-440d-aa29-3b8fa28a5339" (UID: "81e33848-a5b5-440d-aa29-3b8fa28a5339"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:58.512457 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.512437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81e33848-a5b5-440d-aa29-3b8fa28a5339" (UID: "81e33848-a5b5-440d-aa29-3b8fa28a5339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:51:58.512600 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.512580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f" (OuterVolumeSpecName: "kube-api-access-v654f") pod "81e33848-a5b5-440d-aa29-3b8fa28a5339" (UID: "81e33848-a5b5-440d-aa29-3b8fa28a5339"). InnerVolumeSpecName "kube-api-access-v654f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:51:58.611949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.611923    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:58.611949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.611944    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81e33848-a5b5-440d-aa29-3b8fa28a5339-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:58.611949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.611954    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v654f\" (UniqueName: \"kubernetes.io/projected/81e33848-a5b5-440d-aa29-3b8fa28a5339-kube-api-access-v654f\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:58.612125 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:58.611963    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e33848-a5b5-440d-aa29-3b8fa28a5339-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.338021 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.337986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk" event={"ID":"81e33848-a5b5-440d-aa29-3b8fa28a5339","Type":"ContainerDied","Data":"6fee5b17e43ccf1df8db8ac88304bc0d45b3e0d4430b2b52cfdebe0c5c0e94d5"}
Jul 16 17:51:59.338396 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.338034    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"
Jul 16 17:51:59.338396 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.338047    2577 scope.go:117] "RemoveContainer" containerID="ebc815df88864e0d057be0f67e09ccf02a942e9aa3ac8bdb370c53163800fcc3"
Jul 16 17:51:59.347956 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.347937    2577 scope.go:117] "RemoveContainer" containerID="15d8b09ebed29d8cbf383c4d51019b73374f8901b71c0d9f00536d0f51877b1a"
Jul 16 17:51:59.363567 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.363545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:51:59.368846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:51:59.368824    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f46874798-5nvfk"]
Jul 16 17:52:00.305301 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:00.305261    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" path="/var/lib/kubelet/pods/81e33848-a5b5-440d-aa29-3b8fa28a5339/volumes"
Jul 16 17:52:07.395146 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395388    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="storage-initializer"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395398    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="storage-initializer"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395418    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395424    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395438    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395444    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395494    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e33848-a5b5-440d-aa29-3b8fa28a5339" containerName="main"
Jul 16 17:52:07.395521 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395521    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3685774b-11b2-49d5-9395-84ad14bd1823" containerName="main"
Jul 16 17:52:07.395880 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.395531    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3117f868-22c8-46e2-97a5-b2966c47e40d" containerName="main"
Jul 16 17:52:07.398292 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.398274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.401840 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.401818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 16 17:52:07.401954 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.401869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:52:07.409271 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.409247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:52:07.435086 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.435056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:07.481404 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.481580 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.481580 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc9s\" (UniqueName: \"kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.481691 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.481691 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.481691 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.481662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582226 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc9s\" (UniqueName: \"kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582723 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582778 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.582853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.582837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.584481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.584462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.584767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.584751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.594087 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.594067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc9s\" (UniqueName: \"kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.679984 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.679895    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:52:07.683314 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.683291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.686225 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.686201    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-vm76l\""
Jul 16 17:52:07.697109 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.697088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:52:07.709788 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.709769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:07.721488 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.721461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:07.784491 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.784463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.784641 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.784529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7d9\" (UniqueName: \"kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.784702 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.784675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.845167 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.845106    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:52:07.847337 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:52:07.847312    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdac3061_582a_4352_ae4c_baf7ddf7d14c.slice/crio-b32b52a062f1d4df3801036009d1c86ea2b0956ea5e02e5a099f625396914ba8 WatchSource:0}: Error finding container b32b52a062f1d4df3801036009d1c86ea2b0956ea5e02e5a099f625396914ba8: Status 404 returned error can't find the container with id b32b52a062f1d4df3801036009d1c86ea2b0956ea5e02e5a099f625396914ba8
Jul 16 17:52:07.885413 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.885392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.885523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.885431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7d9\" (UniqueName: \"kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.885523 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.885490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.885890 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.885871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.887687 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.887664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.896623 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.896602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7d9\" (UniqueName: \"kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:07.992748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:07.992683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:08.128656 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.128637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:52:08.131088 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:52:08.131063    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6228f8a5_e12b_4543_a545_6bca5f3eca78.slice/crio-5ede0d451032dfe83edb5fe6253d5728bf455ea4f73bf733e5c0c999490a06f5 WatchSource:0}: Error finding container 5ede0d451032dfe83edb5fe6253d5728bf455ea4f73bf733e5c0c999490a06f5: Status 404 returned error can't find the container with id 5ede0d451032dfe83edb5fe6253d5728bf455ea4f73bf733e5c0c999490a06f5
Jul 16 17:52:08.384726 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.384668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" event={"ID":"fdac3061-582a-4352-ae4c-baf7ddf7d14c","Type":"ContainerStarted","Data":"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"}
Jul 16 17:52:08.384896 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.384748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" event={"ID":"fdac3061-582a-4352-ae4c-baf7ddf7d14c","Type":"ContainerStarted","Data":"b32b52a062f1d4df3801036009d1c86ea2b0956ea5e02e5a099f625396914ba8"}
Jul 16 17:52:08.386113 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.386090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" event={"ID":"6228f8a5-e12b-4543-a545-6bca5f3eca78","Type":"ContainerStarted","Data":"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"}
Jul 16 17:52:08.386209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.386127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" event={"ID":"6228f8a5-e12b-4543-a545-6bca5f3eca78","Type":"ContainerStarted","Data":"5ede0d451032dfe83edb5fe6253d5728bf455ea4f73bf733e5c0c999490a06f5"}
Jul 16 17:52:08.386209 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.386144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:08.408885 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.408817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" podStartSLOduration=1.408797841 podStartE2EDuration="1.408797841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:08.405879761 +0000 UTC m=+1055.162287861" watchObservedRunningTime="2026-07-16 17:52:08.408797841 +0000 UTC m=+1055.165205907"
Jul 16 17:52:08.710567 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:08.710478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:17.710650 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:17.710611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:17.722926 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:17.722899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:17.744896 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:17.744852    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" podStartSLOduration=10.744836748 podStartE2EDuration="10.744836748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:08.427666544 +0000 UTC m=+1055.184074618" watchObservedRunningTime="2026-07-16 17:52:17.744836748 +0000 UTC m=+1064.501244810"
Jul 16 17:52:18.446583 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:18.446556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:52:18.916767 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:18.916734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:19.200635 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:19.200558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:31.520684 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:31.520608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:38.512345 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:38.512301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:38.599216 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:38.599183    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:52:39.392586 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:39.392558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:52:49.576517 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:52:49.576457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:02.292166 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.292137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:02.531098 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.531071    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:02.534177 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.534155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.538438 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.538419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 16 17:53:02.538438 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.538420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-xwrgm\""
Jul 16 17:53:02.547529 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.547462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:02.578168 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.578141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:02.618816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.618790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.618908 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.618823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.618908 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.618840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mzg\" (UniqueName: \"kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.719932 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.719905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.720032 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.719936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.720032 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.719952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mzg\" (UniqueName: \"kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.720280 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.720261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.722187 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.722169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.743770 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.743742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mzg\" (UniqueName: \"kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.843164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.843097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:02.974848 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:02.974823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:02.976125 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:53:02.976095    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e00c33_f136_4316_b0b8_8519ead7c092.slice/crio-10cee057c22ae2b0105450684b5a4ab62fdcff573545a5940df2eec1c990574e WatchSource:0}: Error finding container 10cee057c22ae2b0105450684b5a4ab62fdcff573545a5940df2eec1c990574e: Status 404 returned error can't find the container with id 10cee057c22ae2b0105450684b5a4ab62fdcff573545a5940df2eec1c990574e
Jul 16 17:53:03.616304 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:03.616270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" event={"ID":"35e00c33-f136-4316-b0b8-8519ead7c092","Type":"ContainerStarted","Data":"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"}
Jul 16 17:53:03.616815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:03.616325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" event={"ID":"35e00c33-f136-4316-b0b8-8519ead7c092","Type":"ContainerStarted","Data":"10cee057c22ae2b0105450684b5a4ab62fdcff573545a5940df2eec1c990574e"}
Jul 16 17:53:03.616815 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:03.616440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:03.648397 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:03.648338    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" podStartSLOduration=1.648316954 podStartE2EDuration="1.648316954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:53:03.645859894 +0000 UTC m=+1110.402267959" watchObservedRunningTime="2026-07-16 17:53:03.648316954 +0000 UTC m=+1110.404725016"
Jul 16 17:53:15.323353 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:15.323322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:34.622698 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:34.622663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:36.030848 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:36.030811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:36.041565 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:36.041539    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:36.133896 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:36.133863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:44.622356 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:44.622317    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:53:51.068025 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.067958    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" containerName="main" containerID="cri-o://fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711" gracePeriod=45
Jul 16 17:53:51.319984 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.319931    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:51.401122 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.401097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46mzg\" (UniqueName: \"kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg\") pod \"35e00c33-f136-4316-b0b8-8519ead7c092\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") "
Jul 16 17:53:51.401272 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.401134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs\") pod \"35e00c33-f136-4316-b0b8-8519ead7c092\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") "
Jul 16 17:53:51.401272 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.401179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds\") pod \"35e00c33-f136-4316-b0b8-8519ead7c092\" (UID: \"35e00c33-f136-4316-b0b8-8519ead7c092\") "
Jul 16 17:53:51.401497 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.401479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "35e00c33-f136-4316-b0b8-8519ead7c092" (UID: "35e00c33-f136-4316-b0b8-8519ead7c092"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:53:51.403205 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.403181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35e00c33-f136-4316-b0b8-8519ead7c092" (UID: "35e00c33-f136-4316-b0b8-8519ead7c092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:53:51.403301 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.403244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg" (OuterVolumeSpecName: "kube-api-access-46mzg") pod "35e00c33-f136-4316-b0b8-8519ead7c092" (UID: "35e00c33-f136-4316-b0b8-8519ead7c092"). InnerVolumeSpecName "kube-api-access-46mzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:53:51.502083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.502057    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mzg\" (UniqueName: \"kubernetes.io/projected/35e00c33-f136-4316-b0b8-8519ead7c092-kube-api-access-46mzg\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:53:51.502083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.502081    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e00c33-f136-4316-b0b8-8519ead7c092-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:53:51.502083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.502091    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35e00c33-f136-4316-b0b8-8519ead7c092-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:53:51.819409 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.819374    2577 generic.go:356] "Generic (PLEG): container finished" podID="35e00c33-f136-4316-b0b8-8519ead7c092" containerID="fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711" exitCode=0
Jul 16 17:53:51.819604 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.819418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" event={"ID":"35e00c33-f136-4316-b0b8-8519ead7c092","Type":"ContainerDied","Data":"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"}
Jul 16 17:53:51.819604 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.819459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn" event={"ID":"35e00c33-f136-4316-b0b8-8519ead7c092","Type":"ContainerDied","Data":"10cee057c22ae2b0105450684b5a4ab62fdcff573545a5940df2eec1c990574e"}
Jul 16 17:53:51.819604 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.819478    2577 scope.go:117] "RemoveContainer" containerID="fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"
Jul 16 17:53:51.819604 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.819423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"
Jul 16 17:53:51.827954 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.827939    2577 scope.go:117] "RemoveContainer" containerID="fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"
Jul 16 17:53:51.828208 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:53:51.828189    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711\": container with ID starting with fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711 not found: ID does not exist" containerID="fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"
Jul 16 17:53:51.828263 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.828215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711"} err="failed to get container status \"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711\": rpc error: code = NotFound desc = could not find container \"fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711\": container with ID starting with fba68a8ae4d629edbae7e4c46a9d445d5955310a93196ee0bf9a8f5e3c040711 not found: ID does not exist"
Jul 16 17:53:51.850423 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.850398    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:51.862431 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:51.862412    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebgjvn"]
Jul 16 17:53:52.304711 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:52.304679    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" path="/var/lib/kubelet/pods/35e00c33-f136-4316-b0b8-8519ead7c092/volumes"
Jul 16 17:53:56.225174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:56.225100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:53:56.231095 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:56.231052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:56.231460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:56.231439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:53:56.317430 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:56.317400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:53:59.392959 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:53:59.392917    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:54:09.392864 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:09.392831    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:54:11.246070 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.246025    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main" containerID="cri-o://9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df" gracePeriod=45
Jul 16 17:54:11.250636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.250595    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerName="main" containerID="cri-o://43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8" gracePeriod=45
Jul 16 17:54:11.436353 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.436313    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 16 17:54:11.511032 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.511007    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:54:11.513899 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.513881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:54:11.550249 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550343 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfc9s\" (UniqueName: \"kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550384 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv7d9\" (UniqueName: \"kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9\") pod \"6228f8a5-e12b-4543-a545-6bca5f3eca78\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") "
Jul 16 17:54:11.550384 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs\") pod \"6228f8a5-e12b-4543-a545-6bca5f3eca78\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") "
Jul 16 17:54:11.550485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:11.550485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home\") pod \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\" (UID: \"fdac3061-582a-4352-ae4c-baf7ddf7d14c\") "
Jul 16 17:54:11.550713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds\") pod \"6228f8a5-e12b-4543-a545-6bca5f3eca78\" (UID: \"6228f8a5-e12b-4543-a545-6bca5f3eca78\") "
Jul 16 17:54:11.550713 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:11.550868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550830    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.550868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550847    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.550972 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home" (OuterVolumeSpecName: "home") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:11.550972 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.550941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6228f8a5-e12b-4543-a545-6bca5f3eca78" (UID: "6228f8a5-e12b-4543-a545-6bca5f3eca78"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:11.552475 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.552441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9" (OuterVolumeSpecName: "kube-api-access-wv7d9") pod "6228f8a5-e12b-4543-a545-6bca5f3eca78" (UID: "6228f8a5-e12b-4543-a545-6bca5f3eca78"). InnerVolumeSpecName "kube-api-access-wv7d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:54:11.552632 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.552608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:54:11.552939 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.552909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm" (OuterVolumeSpecName: "dshm") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:11.553063 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.553044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6228f8a5-e12b-4543-a545-6bca5f3eca78" (UID: "6228f8a5-e12b-4543-a545-6bca5f3eca78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:54:11.553125 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.553063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s" (OuterVolumeSpecName: "kube-api-access-tfc9s") pod "fdac3061-582a-4352-ae4c-baf7ddf7d14c" (UID: "fdac3061-582a-4352-ae4c-baf7ddf7d14c"). InnerVolumeSpecName "kube-api-access-tfc9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:54:11.652044 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652021    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc9s\" (UniqueName: \"kubernetes.io/projected/fdac3061-582a-4352-ae4c-baf7ddf7d14c-kube-api-access-tfc9s\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652044 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652042    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7d9\" (UniqueName: \"kubernetes.io/projected/6228f8a5-e12b-4543-a545-6bca5f3eca78-kube-api-access-wv7d9\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652051    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f8a5-e12b-4543-a545-6bca5f3eca78-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652063    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdac3061-582a-4352-ae4c-baf7ddf7d14c-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652070    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652077    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdac3061-582a-4352-ae4c-baf7ddf7d14c-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.652169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.652084    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6228f8a5-e12b-4543-a545-6bca5f3eca78-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:54:11.906048 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.905969    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerID="43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8" exitCode=0
Jul 16 17:54:11.906048 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.906019    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"
Jul 16 17:54:11.906219 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.906062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" event={"ID":"fdac3061-582a-4352-ae4c-baf7ddf7d14c","Type":"ContainerDied","Data":"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"}
Jul 16 17:54:11.906219 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.906114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9" event={"ID":"fdac3061-582a-4352-ae4c-baf7ddf7d14c","Type":"ContainerDied","Data":"b32b52a062f1d4df3801036009d1c86ea2b0956ea5e02e5a099f625396914ba8"}
Jul 16 17:54:11.906219 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.906145    2577 scope.go:117] "RemoveContainer" containerID="43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"
Jul 16 17:54:11.907469 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.907446    2577 generic.go:356] "Generic (PLEG): container finished" podID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerID="9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df" exitCode=0
Jul 16 17:54:11.907610 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.907534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" event={"ID":"6228f8a5-e12b-4543-a545-6bca5f3eca78","Type":"ContainerDied","Data":"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"}
Jul 16 17:54:11.907610 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.907561    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"
Jul 16 17:54:11.907610 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.907574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn" event={"ID":"6228f8a5-e12b-4543-a545-6bca5f3eca78","Type":"ContainerDied","Data":"5ede0d451032dfe83edb5fe6253d5728bf455ea4f73bf733e5c0c999490a06f5"}
Jul 16 17:54:11.914856 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.914837    2577 scope.go:117] "RemoveContainer" containerID="43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"
Jul 16 17:54:11.915165 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:54:11.915140    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8\": container with ID starting with 43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8 not found: ID does not exist" containerID="43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"
Jul 16 17:54:11.915255 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.915175    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8"} err="failed to get container status \"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8\": rpc error: code = NotFound desc = could not find container \"43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8\": container with ID starting with 43b3f85f15b41330b1608527c7eef86d8f47d1f7487a290d91c44179e32d15d8 not found: ID does not exist"
Jul 16 17:54:11.915255 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.915198    2577 scope.go:117] "RemoveContainer" containerID="9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"
Jul 16 17:54:11.922980 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.922960    2577 scope.go:117] "RemoveContainer" containerID="9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"
Jul 16 17:54:11.923238 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:54:11.923218    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df\": container with ID starting with 9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df not found: ID does not exist" containerID="9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"
Jul 16 17:54:11.923291 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.923247    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df"} err="failed to get container status \"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df\": rpc error: code = NotFound desc = could not find container \"9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df\": container with ID starting with 9317bcfc55159252190f4ed3c2e00c5770e41f04a6219c1c347eea77bc9c98df not found: ID does not exist"
Jul 16 17:54:11.933694 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.933676    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:54:11.938228 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.938209    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb44dmt9"]
Jul 16 17:54:11.953282 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.953261    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:54:11.959159 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:11.959135    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schels7bn"]
Jul 16 17:54:12.309556 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:12.304911    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" path="/var/lib/kubelet/pods/6228f8a5-e12b-4543-a545-6bca5f3eca78/volumes"
Jul 16 17:54:12.309556 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:12.305541    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" path="/var/lib/kubelet/pods/fdac3061-582a-4352-ae4c-baf7ddf7d14c/volumes"
Jul 16 17:54:37.154077 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154045    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154447    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main"
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154465    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main"
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154477    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" containerName="main"
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154485    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" containerName="main"
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerName="main"
Jul 16 17:54:37.154554 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154528    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerName="main"
Jul 16 17:54:37.154868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154628    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e00c33-f136-4316-b0b8-8519ead7c092" containerName="main"
Jul 16 17:54:37.154868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6228f8a5-e12b-4543-a545-6bca5f3eca78" containerName="main"
Jul 16 17:54:37.154868 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.154654    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdac3061-582a-4352-ae4c-baf7ddf7d14c" containerName="main"
Jul 16 17:54:37.158736 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.158710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.172344 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.172319    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:54:37.172462 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.172328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 16 17:54:37.181354 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.181331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:54:37.201411 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.201390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:54:37.254846 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.254813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.255001 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.254858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.255001 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.254923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.255001 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.254945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l748p\" (UniqueName: \"kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.255001 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.254987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.255147 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.255054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355595 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l748p\" (UniqueName: \"kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.355822 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.355755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.356144 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.356122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.356215 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.356153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.356215 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.356162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.357900 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.357881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.358064 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.358047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.365636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.365615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l748p\" (UniqueName: \"kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.468636 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.468550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:37.519871 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.519836    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:54:37.523797 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.523761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.528360 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.528336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-h69gw\""
Jul 16 17:54:37.542641 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.542618    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:54:37.546021 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.545997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:54:37.613403 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.613369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:54:37.616654 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:54:37.616610    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbfbba5_5fd7_4646_8a81_5e1bf3bf4762.slice/crio-6471dfb7ba82300f1f53695b8bce00c555c5bb110bd0a7971e13eff556ad2074 WatchSource:0}: Error finding container 6471dfb7ba82300f1f53695b8bce00c555c5bb110bd0a7971e13eff556ad2074: Status 404 returned error can't find the container with id 6471dfb7ba82300f1f53695b8bce00c555c5bb110bd0a7971e13eff556ad2074
Jul 16 17:54:37.658681 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.658654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvqq\" (UniqueName: \"kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.658829 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.658806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.658901 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.658845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.760265 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.760234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.760460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.760283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.760460 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.760344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvqq\" (UniqueName: \"kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.760831 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.760809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.762708 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.762686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.779295 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.779269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvqq\" (UniqueName: \"kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.837468 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.837434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:37.969103 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:37.969075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:54:37.971381 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:54:37.971353    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af3a74e_05a3_40ca_bd35_186261766b24.slice/crio-9bd41d96161d852b61c15f95d7d78b2aa4d70d5c785c83c51f67ec49873c7a6a WatchSource:0}: Error finding container 9bd41d96161d852b61c15f95d7d78b2aa4d70d5c785c83c51f67ec49873c7a6a: Status 404 returned error can't find the container with id 9bd41d96161d852b61c15f95d7d78b2aa4d70d5c785c83c51f67ec49873c7a6a
Jul 16 17:54:38.015848 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:38.015782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" event={"ID":"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762","Type":"ContainerStarted","Data":"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"}
Jul 16 17:54:38.015963 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:38.015845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" event={"ID":"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762","Type":"ContainerStarted","Data":"6471dfb7ba82300f1f53695b8bce00c555c5bb110bd0a7971e13eff556ad2074"}
Jul 16 17:54:38.016851 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:38.016828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" event={"ID":"5af3a74e-05a3-40ca-bd35-186261766b24","Type":"ContainerStarted","Data":"9bd41d96161d852b61c15f95d7d78b2aa4d70d5c785c83c51f67ec49873c7a6a"}
Jul 16 17:54:38.042854 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:38.042810    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" podStartSLOduration=1.042795829 podStartE2EDuration="1.042795829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:54:38.041328941 +0000 UTC m=+1204.797737030" watchObservedRunningTime="2026-07-16 17:54:38.042795829 +0000 UTC m=+1204.799203890"
Jul 16 17:54:38.469074 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:38.468998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:39.022276 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:39.022242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" event={"ID":"5af3a74e-05a3-40ca-bd35-186261766b24","Type":"ContainerStarted","Data":"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"}
Jul 16 17:54:39.022418 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:39.022399    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:54:39.051583 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:39.051533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" podStartSLOduration=2.051514091 podStartE2EDuration="2.051514091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:54:39.050922605 +0000 UTC m=+1205.807330666" watchObservedRunningTime="2026-07-16 17:54:39.051514091 +0000 UTC m=+1205.807922148"
Jul 16 17:54:47.468731 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:47.468690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:47.481624 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:47.481601    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:48.068587 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:48.068560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:54:48.149650 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:48.149614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:54:49.750996 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:49.750942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:54:49.847924 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:54:49.847891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:55:10.028761 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:55:10.028729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:55:15.837949 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:55:15.837913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:55:16.122552 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:55:16.122421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:55:46.061434 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:55:46.061403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:55:49.395568 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:55:49.395523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:02.708398 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.708347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:02.975235 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.975157    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:02.978445 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.978425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:02.982537 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.982517    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-wqqk2\""
Jul 16 17:56:02.982643 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.982535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 16 17:56:02.988431 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:02.988410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:03.006890 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.006870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:03.126832 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.126803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xskc\" (UniqueName: \"kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.126977 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.126855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.126977 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.126905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.227816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.227727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xskc\" (UniqueName: \"kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.227816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.227783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.227816 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.227809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.228112 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.228099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.230261 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.230232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.240373 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.240351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xskc\" (UniqueName: \"kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.288221 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.288198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:03.417008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:03.416983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:03.419004 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:56:03.418979    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e34a7a3_4a2f_4227_9241_f1fc7193bd24.slice/crio-35eb08276234821ad60a36fd1db3737fec6b6bc90e92886223b79f46ce589534 WatchSource:0}: Error finding container 35eb08276234821ad60a36fd1db3737fec6b6bc90e92886223b79f46ce589534: Status 404 returned error can't find the container with id 35eb08276234821ad60a36fd1db3737fec6b6bc90e92886223b79f46ce589534
Jul 16 17:56:04.365853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:04.365814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" event={"ID":"0e34a7a3-4a2f-4227-9241-f1fc7193bd24","Type":"ContainerStarted","Data":"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"}
Jul 16 17:56:04.366222 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:04.365869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" event={"ID":"0e34a7a3-4a2f-4227-9241-f1fc7193bd24","Type":"ContainerStarted","Data":"35eb08276234821ad60a36fd1db3737fec6b6bc90e92886223b79f46ce589534"}
Jul 16 17:56:04.366222 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:04.365917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:04.401018 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:04.400944    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" podStartSLOduration=2.400928328 podStartE2EDuration="2.400928328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:04.399435265 +0000 UTC m=+1291.155843330" watchObservedRunningTime="2026-07-16 17:56:04.400928328 +0000 UTC m=+1291.157336389"
Jul 16 17:56:19.731838 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:19.731768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:35.371748 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:35.371721    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:36.978535 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:36.978490    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:37.014718 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:37.014691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:37.077626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:37.077592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:45.371911 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:45.371871    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:56:51.999677 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:51.999629    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerName="main" containerID="cri-o://6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd" gracePeriod=45
Jul 16 17:56:52.240284 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.240259    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:52.330957 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.330876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs\") pod \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") "
Jul 16 17:56:52.330957 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.330926    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xskc\" (UniqueName: \"kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc\") pod \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") "
Jul 16 17:56:52.331164 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.330964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds\") pod \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\" (UID: \"0e34a7a3-4a2f-4227-9241-f1fc7193bd24\") "
Jul 16 17:56:52.331311 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.331281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0e34a7a3-4a2f-4227-9241-f1fc7193bd24" (UID: "0e34a7a3-4a2f-4227-9241-f1fc7193bd24"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:56:52.332986 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.332959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e34a7a3-4a2f-4227-9241-f1fc7193bd24" (UID: "0e34a7a3-4a2f-4227-9241-f1fc7193bd24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:56:52.333065 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.332985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc" (OuterVolumeSpecName: "kube-api-access-2xskc") pod "0e34a7a3-4a2f-4227-9241-f1fc7193bd24" (UID: "0e34a7a3-4a2f-4227-9241-f1fc7193bd24"). InnerVolumeSpecName "kube-api-access-2xskc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:56:52.431983 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.431950    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:56:52.431983 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.431985    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xskc\" (UniqueName: \"kubernetes.io/projected/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-kube-api-access-2xskc\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:56:52.432181 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.432000    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0e34a7a3-4a2f-4227-9241-f1fc7193bd24-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:56:52.562598 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.562567    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerID="6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd" exitCode=0
Jul 16 17:56:52.562764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.562609    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"
Jul 16 17:56:52.562764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.562658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" event={"ID":"0e34a7a3-4a2f-4227-9241-f1fc7193bd24","Type":"ContainerDied","Data":"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"}
Jul 16 17:56:52.562764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.562706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf" event={"ID":"0e34a7a3-4a2f-4227-9241-f1fc7193bd24","Type":"ContainerDied","Data":"35eb08276234821ad60a36fd1db3737fec6b6bc90e92886223b79f46ce589534"}
Jul 16 17:56:52.562764 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.562735    2577 scope.go:117] "RemoveContainer" containerID="6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"
Jul 16 17:56:52.572083 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.572067    2577 scope.go:117] "RemoveContainer" containerID="6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"
Jul 16 17:56:52.572330 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:56:52.572312    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd\": container with ID starting with 6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd not found: ID does not exist" containerID="6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"
Jul 16 17:56:52.572392 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.572338    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd"} err="failed to get container status \"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd\": rpc error: code = NotFound desc = could not find container \"6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd\": container with ID starting with 6809077b3e5704f6c37de5f92ff731567a011faf714a64d0139c9553e3fbb1bd not found: ID does not exist"
Jul 16 17:56:52.587469 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.587402    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:52.588880 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:52.588861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecr5jf"]
Jul 16 17:56:54.305599 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:54.305566    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" path="/var/lib/kubelet/pods/0e34a7a3-4a2f-4227-9241-f1fc7193bd24/volumes"
Jul 16 17:56:57.599450 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:57.599376    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:56:57.610490 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:57.610467    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:56:57.620320 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:57.620294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:56:57.681049 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:56:57.681016    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:00.028784 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:00.028740    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:57:10.028916 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:10.028876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:57:12.619753 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.619698    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" podUID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" containerName="main" containerID="cri-o://e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa" gracePeriod=45
Jul 16 17:57:12.631295 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.631244    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main" containerID="cri-o://5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948" gracePeriod=45
Jul 16 17:57:12.875368 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.875309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:57:12.892116 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892131    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l748p\" (UniqueName: \"kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892437 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:12.892437 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm\") pod \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\" (UID: \"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762\") "
Jul 16 17:57:12.892569 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home" (OuterVolumeSpecName: "home") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:12.892569 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:12.892675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892662    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.892725 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892675    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.892725 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.892687    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.894304 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.894282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm" (OuterVolumeSpecName: "dshm") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:12.894396 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.894329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:57:12.894396 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.894376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p" (OuterVolumeSpecName: "kube-api-access-l748p") pod "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" (UID: "0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762"). InnerVolumeSpecName "kube-api-access-l748p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:57:12.896000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.895981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:57:12.992931 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.992894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs\") pod \"5af3a74e-05a3-40ca-bd35-186261766b24\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") "
Jul 16 17:57:12.992931 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.992939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qvqq\" (UniqueName: \"kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq\") pod \"5af3a74e-05a3-40ca-bd35-186261766b24\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") "
Jul 16 17:57:12.993119 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.993012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds\") pod \"5af3a74e-05a3-40ca-bd35-186261766b24\" (UID: \"5af3a74e-05a3-40ca-bd35-186261766b24\") "
Jul 16 17:57:12.993160 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.993139    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l748p\" (UniqueName: \"kubernetes.io/projected/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-kube-api-access-l748p\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.993160 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.993153    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.993227 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.993160    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:12.993363 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.993324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5af3a74e-05a3-40ca-bd35-186261766b24" (UID: "5af3a74e-05a3-40ca-bd35-186261766b24"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:12.994942 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.994925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5af3a74e-05a3-40ca-bd35-186261766b24" (UID: "5af3a74e-05a3-40ca-bd35-186261766b24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:57:12.995022 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:12.994978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq" (OuterVolumeSpecName: "kube-api-access-2qvqq") pod "5af3a74e-05a3-40ca-bd35-186261766b24" (UID: "5af3a74e-05a3-40ca-bd35-186261766b24"). InnerVolumeSpecName "kube-api-access-2qvqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:57:13.094461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.094428    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5af3a74e-05a3-40ca-bd35-186261766b24-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:13.094461 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.094459    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af3a74e-05a3-40ca-bd35-186261766b24-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:13.094611 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.094473    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qvqq\" (UniqueName: \"kubernetes.io/projected/5af3a74e-05a3-40ca-bd35-186261766b24-kube-api-access-2qvqq\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:57:13.654446 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.654415    2577 generic.go:356] "Generic (PLEG): container finished" podID="5af3a74e-05a3-40ca-bd35-186261766b24" containerID="5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948" exitCode=0
Jul 16 17:57:13.654807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.654462    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"
Jul 16 17:57:13.654807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.654523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" event={"ID":"5af3a74e-05a3-40ca-bd35-186261766b24","Type":"ContainerDied","Data":"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"}
Jul 16 17:57:13.654807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.654569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb" event={"ID":"5af3a74e-05a3-40ca-bd35-186261766b24","Type":"ContainerDied","Data":"9bd41d96161d852b61c15f95d7d78b2aa4d70d5c785c83c51f67ec49873c7a6a"}
Jul 16 17:57:13.654807 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.654590    2577 scope.go:117] "RemoveContainer" containerID="5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"
Jul 16 17:57:13.656000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.655817    2577 generic.go:356] "Generic (PLEG): container finished" podID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" containerID="e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa" exitCode=0
Jul 16 17:57:13.656000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.655872    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"
Jul 16 17:57:13.656000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.655901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" event={"ID":"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762","Type":"ContainerDied","Data":"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"}
Jul 16 17:57:13.656000 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.655949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5" event={"ID":"0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762","Type":"ContainerDied","Data":"6471dfb7ba82300f1f53695b8bce00c555c5bb110bd0a7971e13eff556ad2074"}
Jul 16 17:57:13.662719 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.662704    2577 scope.go:117] "RemoveContainer" containerID="5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"
Jul 16 17:57:13.662953 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:57:13.662935    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948\": container with ID starting with 5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948 not found: ID does not exist" containerID="5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"
Jul 16 17:57:13.663024 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.662966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948"} err="failed to get container status \"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948\": rpc error: code = NotFound desc = could not find container \"5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948\": container with ID starting with 5c149f4b010543d8114ba2096b6ace3ebc3f98eabbc3b9f8fe0a2e84377f4948 not found: ID does not exist"
Jul 16 17:57:13.663024 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.662988    2577 scope.go:117] "RemoveContainer" containerID="e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"
Jul 16 17:57:13.673709 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.673690    2577 scope.go:117] "RemoveContainer" containerID="e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"
Jul 16 17:57:13.673961 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:57:13.673940    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa\": container with ID starting with e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa not found: ID does not exist" containerID="e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"
Jul 16 17:57:13.674005 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.673968    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa"} err="failed to get container status \"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa\": rpc error: code = NotFound desc = could not find container \"e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa\": container with ID starting with e9637518fb6a57fdb5fadb2772493c26359174df57f97f9f2a1efdafe33909fa not found: ID does not exist"
Jul 16 17:57:13.689369 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.689346    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:57:13.698242 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.698221    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vcqb5"]
Jul 16 17:57:13.716724 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.716693    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:57:13.721287 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:13.721269    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep6gtb"]
Jul 16 17:57:14.304787 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:14.304756    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" path="/var/lib/kubelet/pods/0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762/volumes"
Jul 16 17:57:14.305169 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:14.305147    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" path="/var/lib/kubelet/pods/5af3a74e-05a3-40ca-bd35-186261766b24/volumes"
Jul 16 17:57:23.089308 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.089258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:23.379876 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.379795    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:57:23.380236 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380219    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main"
Jul 16 17:57:23.380305 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380240    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main"
Jul 16 17:57:23.380305 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380269    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerName="main"
Jul 16 17:57:23.380305 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380277    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerName="main"
Jul 16 17:57:23.380305 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" containerName="main"
Jul 16 17:57:23.380433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" containerName="main"
Jul 16 17:57:23.380433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380386    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbfbba5-5fd7-4646-8a81-5e1bf3bf4762" containerName="main"
Jul 16 17:57:23.380433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e34a7a3-4a2f-4227-9241-f1fc7193bd24" containerName="main"
Jul 16 17:57:23.380433 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.380405    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af3a74e-05a3-40ca-bd35-186261766b24" containerName="main"
Jul 16 17:57:23.384897 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.384880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.388123 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.388104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 16 17:57:23.388207 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.388127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:57:23.388259 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.388114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-gpzlf\""
Jul 16 17:57:23.388259 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.388114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:57:23.393678 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.393660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:57:23.413694 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.413672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:23.477879 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.477853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5f72\" (UniqueName: \"kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.477967 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.477890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.478008 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.477985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.478044 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.478025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579325 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5f72\" (UniqueName: \"kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579427 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579791 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.579791 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.579780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.581737 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.581720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.589223 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.589201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5f72\" (UniqueName: \"kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.694344 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.694282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:23.826907 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:23.826878    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:57:23.830408 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:57:23.830383    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113ef214_4aad_40fc_935f_b813096b8b75.slice/crio-e3fb92b9a09be25bd34077b7321ab1185d6cd2e0041624abe934eb1446af9af0 WatchSource:0}: Error finding container e3fb92b9a09be25bd34077b7321ab1185d6cd2e0041624abe934eb1446af9af0: Status 404 returned error can't find the container with id e3fb92b9a09be25bd34077b7321ab1185d6cd2e0041624abe934eb1446af9af0
Jul 16 17:57:24.703029 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:24.702996    2577 generic.go:356] "Generic (PLEG): container finished" podID="113ef214-4aad-40fc-935f-b813096b8b75" containerID="ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4" exitCode=0
Jul 16 17:57:24.703318 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:24.703050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" event={"ID":"113ef214-4aad-40fc-935f-b813096b8b75","Type":"ContainerDied","Data":"ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4"}
Jul 16 17:57:24.703318 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:24.703102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" event={"ID":"113ef214-4aad-40fc-935f-b813096b8b75","Type":"ContainerStarted","Data":"e3fb92b9a09be25bd34077b7321ab1185d6cd2e0041624abe934eb1446af9af0"}
Jul 16 17:57:25.709174 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:25.709133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" event={"ID":"113ef214-4aad-40fc-935f-b813096b8b75","Type":"ContainerStarted","Data":"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"}
Jul 16 17:57:25.709584 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:25.709214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:25.740843 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:25.740799    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" podStartSLOduration=1.988047297 podStartE2EDuration="2.740785003s" totalImagesPullingTime="0s" totalInitContainerRuntime="752.737706ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:25.737635891 +0000 UTC m=+1372.494043953" watchObservedRunningTime="2026-07-16 17:57:25.740785003 +0000 UTC m=+1372.497193064"
Jul 16 17:57:26.039826 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:26.039789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:26.147080 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:26.147044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:37.654178 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:37.654136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:55.872153 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:55.872120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:57:56.716089 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:56.716057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:57:57.126119 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:57:57.126076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:58:05.882640 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.882608    2577 generic.go:356] "Generic (PLEG): container finished" podID="123399d5-de98-4560-b7e3-37814a765e9b" containerID="d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a" exitCode=0
Jul 16 17:58:05.882979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.882680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerDied","Data":"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"}
Jul 16 17:58:05.883026 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.883014    2577 scope.go:117] "RemoveContainer" containerID="d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"
Jul 16 17:58:05.883921 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.883896    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:58:05.883976 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.883957    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:58:05.884010 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:05.883995    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 17:58:06.888749 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:06.888716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerStarted","Data":"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"}
Jul 16 17:58:07.152731 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:07.152639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:58:07.152911 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:07.152877    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 16 17:58:08.153358 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:08.153316    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 16 17:58:09.153016 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:09.152972    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 16 17:58:10.153089 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:10.153040    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 16 17:58:11.152999 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:11.152958    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Jul 16 17:58:12.156958 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:12.156930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:58:12.157330 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:12.157239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:58:12.158243 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:58:12.158226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 17:59:07.121385 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:07.121338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:59:08.456138 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:08.456105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:59:08.473053 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:08.473027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:59:08.531361 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:08.531336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:59:16.715725 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:16.715677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:59:23.493060 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.493025    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="main" containerID="cri-o://684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537" gracePeriod=45
Jul 16 17:59:23.735196 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.735173    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:59:23.854853 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.854826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs\") pod \"113ef214-4aad-40fc-935f-b813096b8b75\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") "
Jul 16 17:59:23.854979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.854871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5f72\" (UniqueName: \"kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72\") pod \"113ef214-4aad-40fc-935f-b813096b8b75\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") "
Jul 16 17:59:23.854979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.854904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds\") pod \"113ef214-4aad-40fc-935f-b813096b8b75\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") "
Jul 16 17:59:23.854979 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.854941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location\") pod \"113ef214-4aad-40fc-935f-b813096b8b75\" (UID: \"113ef214-4aad-40fc-935f-b813096b8b75\") "
Jul 16 17:59:23.855211 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.855191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "113ef214-4aad-40fc-935f-b813096b8b75" (UID: "113ef214-4aad-40fc-935f-b813096b8b75"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:23.855675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.855646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "113ef214-4aad-40fc-935f-b813096b8b75" (UID: "113ef214-4aad-40fc-935f-b813096b8b75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:23.856874 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.856852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "113ef214-4aad-40fc-935f-b813096b8b75" (UID: "113ef214-4aad-40fc-935f-b813096b8b75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:59:23.856953 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.856901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72" (OuterVolumeSpecName: "kube-api-access-h5f72") pod "113ef214-4aad-40fc-935f-b813096b8b75" (UID: "113ef214-4aad-40fc-935f-b813096b8b75"). InnerVolumeSpecName "kube-api-access-h5f72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:59:23.955630 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.955608    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5f72\" (UniqueName: \"kubernetes.io/projected/113ef214-4aad-40fc-935f-b813096b8b75-kube-api-access-h5f72\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:59:23.955630 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.955629    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:59:23.955746 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.955638    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/113ef214-4aad-40fc-935f-b813096b8b75-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:59:23.955746 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:23.955648    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113ef214-4aad-40fc-935f-b813096b8b75-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.207637 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.207575    2577 generic.go:356] "Generic (PLEG): container finished" podID="113ef214-4aad-40fc-935f-b813096b8b75" containerID="684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537" exitCode=0
Jul 16 17:59:24.207637 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.207633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"
Jul 16 17:59:24.207756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.207657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" event={"ID":"113ef214-4aad-40fc-935f-b813096b8b75","Type":"ContainerDied","Data":"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"}
Jul 16 17:59:24.207756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.207712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4" event={"ID":"113ef214-4aad-40fc-935f-b813096b8b75","Type":"ContainerDied","Data":"e3fb92b9a09be25bd34077b7321ab1185d6cd2e0041624abe934eb1446af9af0"}
Jul 16 17:59:24.207756 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.207739    2577 scope.go:117] "RemoveContainer" containerID="684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"
Jul 16 17:59:24.218546 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.218492    2577 scope.go:117] "RemoveContainer" containerID="ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4"
Jul 16 17:59:24.225959 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.225943    2577 scope.go:117] "RemoveContainer" containerID="684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"
Jul 16 17:59:24.226209 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:59:24.226187    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537\": container with ID starting with 684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537 not found: ID does not exist" containerID="684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"
Jul 16 17:59:24.226264 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.226218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537"} err="failed to get container status \"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537\": rpc error: code = NotFound desc = could not find container \"684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537\": container with ID starting with 684594b3097ee05e9678049bd8b798817a7b267ab8242d66284c2c7238287537 not found: ID does not exist"
Jul 16 17:59:24.226264 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.226235    2577 scope.go:117] "RemoveContainer" containerID="ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4"
Jul 16 17:59:24.226445 ip-10-0-142-99 kubenswrapper[2577]: E0716 17:59:24.226426    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4\": container with ID starting with ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4 not found: ID does not exist" containerID="ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4"
Jul 16 17:59:24.226485 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.226451    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4"} err="failed to get container status \"ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4\": rpc error: code = NotFound desc = could not find container \"ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4\": container with ID starting with ab3d06974731dcd6ffb04516addc77b30eef898af277b54b23a1a9299eb194f4 not found: ID does not exist"
Jul 16 17:59:24.244626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.244603    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:59:24.250163 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.250144    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew77k4"]
Jul 16 17:59:24.304441 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:24.304419    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113ef214-4aad-40fc-935f-b813096b8b75" path="/var/lib/kubelet/pods/113ef214-4aad-40fc-935f-b813096b8b75/volumes"
Jul 16 17:59:53.130987 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.130959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:59:53.566173 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 17:59:53.566431 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566420    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="main"
Jul 16 17:59:53.566476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566432    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="main"
Jul 16 17:59:53.566476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566449    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="storage-initializer"
Jul 16 17:59:53.566476 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566455    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="storage-initializer"
Jul 16 17:59:53.566622 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.566518    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="113ef214-4aad-40fc-935f-b813096b8b75" containerName="main"
Jul 16 17:59:53.570661 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.570644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.573290 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.573267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:59:53.573410 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.573268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 16 17:59:53.574219 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.574199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:59:53.574313 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.574230    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-bqknm\""
Jul 16 17:59:53.581839 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.581819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 17:59:53.593726 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.593702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 17:59:53.694340 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.694315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.694483 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.694373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.694483 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.694467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.694662 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.694494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt8v\" (UniqueName: \"kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.795481 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.795626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.795626 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt8v\" (UniqueName: \"kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.795745 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.795942 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.796033 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.795989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.797833 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.797816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.805061 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.805042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt8v\" (UniqueName: \"kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:53.880577 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:53.880514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:54.009447 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:54.009304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 17:59:54.012007 ip-10-0-142-99 kubenswrapper[2577]: W0716 17:59:54.011982    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0752b0d_9064_4a04_8b9a_5fa9f511e526.slice/crio-0786fb9402c484783b5991024e94d57879a5d406e887e9264698848b2c41cb89 WatchSource:0}: Error finding container 0786fb9402c484783b5991024e94d57879a5d406e887e9264698848b2c41cb89: Status 404 returned error can't find the container with id 0786fb9402c484783b5991024e94d57879a5d406e887e9264698848b2c41cb89
Jul 16 17:59:54.327740 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:54.327699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerStarted","Data":"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"}
Jul 16 17:59:54.328145 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:54.327766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerStarted","Data":"0786fb9402c484783b5991024e94d57879a5d406e887e9264698848b2c41cb89"}
Jul 16 17:59:55.333287 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:55.333251    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerID="99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806" exitCode=0
Jul 16 17:59:55.333675 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:55.333321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerDied","Data":"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"}
Jul 16 17:59:56.339139 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:56.339102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerStarted","Data":"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"}
Jul 16 17:59:56.339619 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:56.339207    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 17:59:56.362024 ip-10-0-142-99 kubenswrapper[2577]: I0716 17:59:56.361972    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" podStartSLOduration=2.613443153 podStartE2EDuration="3.361958746s" totalImagesPullingTime="0s" totalInitContainerRuntime="748.515593ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:59:56.361064293 +0000 UTC m=+1523.117472358" watchObservedRunningTime="2026-07-16 17:59:56.361958746 +0000 UTC m=+1523.118366808"
Jul 16 18:00:26.913703 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:00:26.913662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:00:27.344033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:00:27.344007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 18:00:28.979480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:00:28.979447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:00.760353 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:00.760319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:00.820242 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:00.820214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:25.010829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:25.010784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 18:01:25.019673 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:25.019647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:26.572729 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:26.572698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:27.344919 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:27.344885    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:01:37.344367 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:37.344325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:01:40.034227 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.034175    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main" containerID="cri-o://6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067" gracePeriod=45
Jul 16 18:01:40.285795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.285739    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 18:01:40.430201 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430177    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location\") pod \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") "
Jul 16 18:01:40.430372 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds\") pod \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") "
Jul 16 18:01:40.430372 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430246    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmt8v\" (UniqueName: \"kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v\") pod \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") "
Jul 16 18:01:40.430372 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs\") pod \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\" (UID: \"e0752b0d-9064-4a04-8b9a-5fa9f511e526\") "
Jul 16 18:01:40.430554 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e0752b0d-9064-4a04-8b9a-5fa9f511e526" (UID: "e0752b0d-9064-4a04-8b9a-5fa9f511e526"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.430611 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430568    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.430918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.430884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0752b0d-9064-4a04-8b9a-5fa9f511e526" (UID: "e0752b0d-9064-4a04-8b9a-5fa9f511e526"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.432310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.432281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0752b0d-9064-4a04-8b9a-5fa9f511e526" (UID: "e0752b0d-9064-4a04-8b9a-5fa9f511e526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:01:40.432397 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.432362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v" (OuterVolumeSpecName: "kube-api-access-nmt8v") pod "e0752b0d-9064-4a04-8b9a-5fa9f511e526" (UID: "e0752b0d-9064-4a04-8b9a-5fa9f511e526"). InnerVolumeSpecName "kube-api-access-nmt8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:01:40.531711 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.531690    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.531711 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.531709    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmt8v\" (UniqueName: \"kubernetes.io/projected/e0752b0d-9064-4a04-8b9a-5fa9f511e526-kube-api-access-nmt8v\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.531830 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.531719    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0752b0d-9064-4a04-8b9a-5fa9f511e526-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.749982 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.749955    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerID="6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067" exitCode=0
Jul 16 18:01:40.750098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.750015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"
Jul 16 18:01:40.750098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.750014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerDied","Data":"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"}
Jul 16 18:01:40.750184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.750144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh" event={"ID":"e0752b0d-9064-4a04-8b9a-5fa9f511e526","Type":"ContainerDied","Data":"0786fb9402c484783b5991024e94d57879a5d406e887e9264698848b2c41cb89"}
Jul 16 18:01:40.750184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.750179    2577 scope.go:117] "RemoveContainer" containerID="6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"
Jul 16 18:01:40.759708 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.759690    2577 scope.go:117] "RemoveContainer" containerID="99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"
Jul 16 18:01:40.767650 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.767633    2577 scope.go:117] "RemoveContainer" containerID="6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"
Jul 16 18:01:40.767955 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:01:40.767934    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067\": container with ID starting with 6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067 not found: ID does not exist" containerID="6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"
Jul 16 18:01:40.767996 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.767965    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067"} err="failed to get container status \"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067\": rpc error: code = NotFound desc = could not find container \"6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067\": container with ID starting with 6863126f6810a672bef516cccc020b78b01d14af1d35a079a544484cffe93067 not found: ID does not exist"
Jul 16 18:01:40.767996 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.767983    2577 scope.go:117] "RemoveContainer" containerID="99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"
Jul 16 18:01:40.768248 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:01:40.768228    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806\": container with ID starting with 99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806 not found: ID does not exist" containerID="99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"
Jul 16 18:01:40.768296 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.768255    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806"} err="failed to get container status \"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806\": rpc error: code = NotFound desc = could not find container \"99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806\": container with ID starting with 99abb0bf20a252422212bad44f81b14434032c0ab06109011c1fa4e42fd59806 not found: ID does not exist"
Jul 16 18:01:40.774754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.774728    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 18:01:40.783261 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:40.783240    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel4rnh"]
Jul 16 18:01:42.304396 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:42.304364    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" path="/var/lib/kubelet/pods/e0752b0d-9064-4a04-8b9a-5fa9f511e526/volumes"
Jul 16 18:01:45.347574 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:45.347526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:45.457539 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:45.457490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:45.512429 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:45.512401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:47.876644 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:47.876611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:47.977756 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:47.977723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:01:48.509964 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:01:48.509930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:02:14.103516 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.103482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:02:14.439324 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439291    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:02:14.439767 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439744    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main"
Jul 16 18:02:14.439767 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439764    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main"
Jul 16 18:02:14.439933 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="storage-initializer"
Jul 16 18:02:14.439933 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439811    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="storage-initializer"
Jul 16 18:02:14.439933 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.439875    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0752b0d-9064-4a04-8b9a-5fa9f511e526" containerName="main"
Jul 16 18:02:14.442738 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.442720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.446343 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.446321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rvqnv\""
Jul 16 18:02:14.446436 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.446322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:02:14.446436 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.446332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 16 18:02:14.456749 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.456725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:02:14.465661 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.465628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:02:14.480886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.480863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.480972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.480909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4gd\" (UniqueName: \"kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.480972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.480938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.582361 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.582329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.582493 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.582397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4gd\" (UniqueName: \"kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.582493 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.582440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.582880 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.582854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.584706 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.584686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.593903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.593882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4gd\" (UniqueName: \"kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.755572 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.755549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:14.879703 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:14.879682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:02:14.881942 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:02:14.881914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cba3a1b_17cd_4a78_b09f_9531cff003d7.slice/crio-f347a4a8b0c3ce9034f59dc9d1e43aba712762f99a73c5b045402dbf83406d71 WatchSource:0}: Error finding container f347a4a8b0c3ce9034f59dc9d1e43aba712762f99a73c5b045402dbf83406d71: Status 404 returned error can't find the container with id f347a4a8b0c3ce9034f59dc9d1e43aba712762f99a73c5b045402dbf83406d71
Jul 16 18:02:15.884062 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:15.884019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" event={"ID":"6cba3a1b-17cd-4a78-b09f-9531cff003d7","Type":"ContainerStarted","Data":"45c7fcc07619ec62cc04df2e92aab1b688b45ee75fac7cc258214dbedda2c077"}
Jul 16 18:02:15.884453 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:15.884083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" event={"ID":"6cba3a1b-17cd-4a78-b09f-9531cff003d7","Type":"ContainerStarted","Data":"f347a4a8b0c3ce9034f59dc9d1e43aba712762f99a73c5b045402dbf83406d71"}
Jul 16 18:02:15.884453 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:15.884107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:15.906210 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:15.906154    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" podStartSLOduration=1.9061402589999998 podStartE2EDuration="1.906140259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:02:15.905436169 +0000 UTC m=+1662.661844234" watchObservedRunningTime="2026-07-16 18:02:15.906140259 +0000 UTC m=+1662.662548320"
Jul 16 18:02:29.329663 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:29.329623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:02:46.890359 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:46.890321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:02:56.606175 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:02:56.606106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:03:30.770176 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:30.770141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:03:31.267988 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.267958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:03:31.271244 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.271220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.273890 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.273861    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-pbhjp\""
Jul 16 18:03:31.274015 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.273908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:03:31.274015 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.273908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 16 18:03:31.283653 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.283633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:03:31.304815 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.304792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:03:31.378603 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.378578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wqg\" (UniqueName: \"kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.378704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.378672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.378754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.378715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.378754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.378738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.479583 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.479555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wqg\" (UniqueName: \"kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.479706 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.479604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.479706 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.479636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.479706 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.479666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.480032 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.480011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.480100 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.480042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.481960 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.481944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.489016 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.488990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wqg\" (UniqueName: \"kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.580952 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.580895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:31.719136 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:31.719102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:03:31.721581 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:03:31.721555    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701fb989_7ee7_4bcd_a011_11ce88030498.slice/crio-c29937a03ccd0d079fa34475fd2d3d85fb6aaf5e30ba5763ae777c7cfeab421d WatchSource:0}: Error finding container c29937a03ccd0d079fa34475fd2d3d85fb6aaf5e30ba5763ae777c7cfeab421d: Status 404 returned error can't find the container with id c29937a03ccd0d079fa34475fd2d3d85fb6aaf5e30ba5763ae777c7cfeab421d
Jul 16 18:03:32.194457 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:32.194414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerStarted","Data":"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"}
Jul 16 18:03:32.194829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:32.194478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerStarted","Data":"c29937a03ccd0d079fa34475fd2d3d85fb6aaf5e30ba5763ae777c7cfeab421d"}
Jul 16 18:03:33.199431 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:33.199399    2577 generic.go:356] "Generic (PLEG): container finished" podID="701fb989-7ee7-4bcd-a011-11ce88030498" containerID="bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277" exitCode=0
Jul 16 18:03:33.199940 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:33.199486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerDied","Data":"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"}
Jul 16 18:03:34.205118 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:34.205083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerStarted","Data":"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"}
Jul 16 18:03:34.205596 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:34.205193    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:03:34.227102 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:34.227060    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" podStartSLOduration=2.465817933 podStartE2EDuration="3.227046957s" totalImagesPullingTime="0s" totalInitContainerRuntime="761.229024ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:03:34.226053198 +0000 UTC m=+1740.982461297" watchObservedRunningTime="2026-07-16 18:03:34.227046957 +0000 UTC m=+1740.983455016"
Jul 16 18:03:40.125233 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:03:40.125193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:04:05.211294 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:05.211266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:04:27.337546 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:27.337449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:04:27.342341 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:27.342318    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:04:27.424941 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:27.424917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:04:36.890825 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:36.890787    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:04:42.361819 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.361778    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerName="main" containerID="cri-o://45c7fcc07619ec62cc04df2e92aab1b688b45ee75fac7cc258214dbedda2c077" gracePeriod=45
Jul 16 18:04:42.489093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.489054    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerID="45c7fcc07619ec62cc04df2e92aab1b688b45ee75fac7cc258214dbedda2c077" exitCode=0
Jul 16 18:04:42.489227 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.489116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" event={"ID":"6cba3a1b-17cd-4a78-b09f-9531cff003d7","Type":"ContainerDied","Data":"45c7fcc07619ec62cc04df2e92aab1b688b45ee75fac7cc258214dbedda2c077"}
Jul 16 18:04:42.616346 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.616289    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:04:42.742511 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.742484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs\") pod \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") "
Jul 16 18:04:42.742637 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.742551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx4gd\" (UniqueName: \"kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd\") pod \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") "
Jul 16 18:04:42.742637 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.742578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds\") pod \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\" (UID: \"6cba3a1b-17cd-4a78-b09f-9531cff003d7\") "
Jul 16 18:04:42.742914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.742884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6cba3a1b-17cd-4a78-b09f-9531cff003d7" (UID: "6cba3a1b-17cd-4a78-b09f-9531cff003d7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:04:42.744535 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.744490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd" (OuterVolumeSpecName: "kube-api-access-mx4gd") pod "6cba3a1b-17cd-4a78-b09f-9531cff003d7" (UID: "6cba3a1b-17cd-4a78-b09f-9531cff003d7"). InnerVolumeSpecName "kube-api-access-mx4gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:04:42.744650 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.744616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cba3a1b-17cd-4a78-b09f-9531cff003d7" (UID: "6cba3a1b-17cd-4a78-b09f-9531cff003d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:04:42.844105 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.844079    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:04:42.844105 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.844100    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4gd\" (UniqueName: \"kubernetes.io/projected/6cba3a1b-17cd-4a78-b09f-9531cff003d7-kube-api-access-mx4gd\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:04:42.844233 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:42.844110    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cba3a1b-17cd-4a78-b09f-9531cff003d7-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.494700 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:43.494661    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"
Jul 16 18:04:43.494700 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:43.494687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5" event={"ID":"6cba3a1b-17cd-4a78-b09f-9531cff003d7","Type":"ContainerDied","Data":"f347a4a8b0c3ce9034f59dc9d1e43aba712762f99a73c5b045402dbf83406d71"}
Jul 16 18:04:43.495200 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:43.494753    2577 scope.go:117] "RemoveContainer" containerID="45c7fcc07619ec62cc04df2e92aab1b688b45ee75fac7cc258214dbedda2c077"
Jul 16 18:04:43.517190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:43.517158    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:04:43.520389 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:43.520368    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche6qvh5"]
Jul 16 18:04:44.306408 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:44.306373    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" path="/var/lib/kubelet/pods/6cba3a1b-17cd-4a78-b09f-9531cff003d7/volumes"
Jul 16 18:04:57.201550 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.201486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:04:57.442006 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.441974    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:04:57.442261 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.442249    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerName="main"
Jul 16 18:04:57.442305 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.442262    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerName="main"
Jul 16 18:04:57.442340 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.442328    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cba3a1b-17cd-4a78-b09f-9531cff003d7" containerName="main"
Jul 16 18:04:57.446307 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.446290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.449744 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.449724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-6mlc4\""
Jul 16 18:04:57.449870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.449775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 16 18:04:57.454907 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.454888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:04:57.459990 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.459970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:04:57.560578 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.560552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.560719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.560621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt92f\" (UniqueName: \"kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.560719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.560675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.661912 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.661889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.662008 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.661940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.662008 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.661975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt92f\" (UniqueName: \"kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.662258 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.662239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.664351 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.664328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.671351 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.671330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt92f\" (UniqueName: \"kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.755935 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.755912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:57.883989 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:57.883963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:04:57.886128 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:04:57.886100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f54d93a_7e31_4901_9981_e0ffe59cea72.slice/crio-4273b770ae192b00695c89e166a7555b96ee6959cf77d818440ea8b53bba36ea WatchSource:0}: Error finding container 4273b770ae192b00695c89e166a7555b96ee6959cf77d818440ea8b53bba36ea: Status 404 returned error can't find the container with id 4273b770ae192b00695c89e166a7555b96ee6959cf77d818440ea8b53bba36ea
Jul 16 18:04:58.560380 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:58.560339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" event={"ID":"0f54d93a-7e31-4901-9981-e0ffe59cea72","Type":"ContainerStarted","Data":"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"}
Jul 16 18:04:58.560861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:58.560403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" event={"ID":"0f54d93a-7e31-4901-9981-e0ffe59cea72","Type":"ContainerStarted","Data":"4273b770ae192b00695c89e166a7555b96ee6959cf77d818440ea8b53bba36ea"}
Jul 16 18:04:58.560861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:58.560438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:04:58.582451 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:04:58.582406    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" podStartSLOduration=1.582391097 podStartE2EDuration="1.582391097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:04:58.580829503 +0000 UTC m=+1825.337237568" watchObservedRunningTime="2026-07-16 18:04:58.582391097 +0000 UTC m=+1825.338799158"
Jul 16 18:05:16.945935 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:16.945898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:05:24.352624 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:24.352583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:05:24.426905 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:24.426876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:05:29.565823 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:29.565790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:05:35.907248 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:35.907217    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:05:35.923564 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:35.923535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:05:35.986362 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:35.986333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:05:45.211729 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:45.211683    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:05:50.927150 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:50.927103    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="main" containerID="cri-o://516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46" gracePeriod=45
Jul 16 18:05:51.170647 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.170625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:05:51.273836 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.273812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2wqg\" (UniqueName: \"kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg\") pod \"701fb989-7ee7-4bcd-a011-11ce88030498\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") "
Jul 16 18:05:51.273975 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.273878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location\") pod \"701fb989-7ee7-4bcd-a011-11ce88030498\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") "
Jul 16 18:05:51.273975 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.273935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds\") pod \"701fb989-7ee7-4bcd-a011-11ce88030498\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") "
Jul 16 18:05:51.274098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.274010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs\") pod \"701fb989-7ee7-4bcd-a011-11ce88030498\" (UID: \"701fb989-7ee7-4bcd-a011-11ce88030498\") "
Jul 16 18:05:51.274249 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.274209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "701fb989-7ee7-4bcd-a011-11ce88030498" (UID: "701fb989-7ee7-4bcd-a011-11ce88030498"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.274542 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.274520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "701fb989-7ee7-4bcd-a011-11ce88030498" (UID: "701fb989-7ee7-4bcd-a011-11ce88030498"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.275905 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.275883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg" (OuterVolumeSpecName: "kube-api-access-v2wqg") pod "701fb989-7ee7-4bcd-a011-11ce88030498" (UID: "701fb989-7ee7-4bcd-a011-11ce88030498"). InnerVolumeSpecName "kube-api-access-v2wqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:05:51.275984 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.275896    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "701fb989-7ee7-4bcd-a011-11ce88030498" (UID: "701fb989-7ee7-4bcd-a011-11ce88030498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:05:51.375116 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.374865    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.375116 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.374894    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/701fb989-7ee7-4bcd-a011-11ce88030498-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.375116 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.374910    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701fb989-7ee7-4bcd-a011-11ce88030498-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.375116 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.374985    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2wqg\" (UniqueName: \"kubernetes.io/projected/701fb989-7ee7-4bcd-a011-11ce88030498-kube-api-access-v2wqg\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.786324 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.786292    2577 generic.go:356] "Generic (PLEG): container finished" podID="701fb989-7ee7-4bcd-a011-11ce88030498" containerID="516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46" exitCode=0
Jul 16 18:05:51.786480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.786344    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"
Jul 16 18:05:51.786480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.786373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerDied","Data":"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"}
Jul 16 18:05:51.786480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.786427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck" event={"ID":"701fb989-7ee7-4bcd-a011-11ce88030498","Type":"ContainerDied","Data":"c29937a03ccd0d079fa34475fd2d3d85fb6aaf5e30ba5763ae777c7cfeab421d"}
Jul 16 18:05:51.786480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.786448    2577 scope.go:117] "RemoveContainer" containerID="516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"
Jul 16 18:05:51.794954 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.794939    2577 scope.go:117] "RemoveContainer" containerID="bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"
Jul 16 18:05:51.802228 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.802214    2577 scope.go:117] "RemoveContainer" containerID="516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"
Jul 16 18:05:51.802457 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:05:51.802439    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46\": container with ID starting with 516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46 not found: ID does not exist" containerID="516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"
Jul 16 18:05:51.802528 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.802465    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46"} err="failed to get container status \"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46\": rpc error: code = NotFound desc = could not find container \"516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46\": container with ID starting with 516c388d87e495fc9c7dcf9db6091d1c7f8ed4c21b3f028613dfa6d4385d8d46 not found: ID does not exist"
Jul 16 18:05:51.802528 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.802482    2577 scope.go:117] "RemoveContainer" containerID="bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"
Jul 16 18:05:51.802710 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:05:51.802695    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277\": container with ID starting with bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277 not found: ID does not exist" containerID="bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"
Jul 16 18:05:51.802751 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.802713    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277"} err="failed to get container status \"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277\": rpc error: code = NotFound desc = could not find container \"bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277\": container with ID starting with bd7d6a94d576fb83938d6687fb930a33d4d47dc5755e7c56ad1d41b17f0e3277 not found: ID does not exist"
Jul 16 18:05:51.809953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.809933    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:05:51.814263 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:51.814243    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche9fnck"]
Jul 16 18:05:52.305069 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:05:52.305038    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" path="/var/lib/kubelet/pods/701fb989-7ee7-4bcd-a011-11ce88030498/volumes"
Jul 16 18:06:12.105139 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.105064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:06:12.383017 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.382945    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:06:12.383304 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.383288    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="storage-initializer"
Jul 16 18:06:12.383387 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.383308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="storage-initializer"
Jul 16 18:06:12.383387 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.383322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="main"
Jul 16 18:06:12.383387 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.383330    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="main"
Jul 16 18:06:12.383563 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.383430    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="701fb989-7ee7-4bcd-a011-11ce88030498" containerName="main"
Jul 16 18:06:12.386444 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.386426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.389920 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.389902    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-vqfxx\""
Jul 16 18:06:12.390015 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.389905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:06:12.390015 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.389905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 16 18:06:12.398219 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.398198    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:06:12.513732 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.513702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:06:12.545861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.545836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.546001 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.545879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.546001 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.545900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqw9\" (UniqueName: \"kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.546090 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.545999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647325 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647325 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647490 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647490 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqw9\" (UniqueName: \"kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647867 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.647867 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.647859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.649792 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.649770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.657429 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.657409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqw9\" (UniqueName: \"kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9\") pod \"custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.695366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.695347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:12.821103 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.821071    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:06:12.824040 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:06:12.824014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00afb7b6_a55b_41e1_9bf3_674f49733fd5.slice/crio-2b2c8d3383a025d435ec2128167ef1ea721b530c9540d3016afc36e25ea90a2d WatchSource:0}: Error finding container 2b2c8d3383a025d435ec2128167ef1ea721b530c9540d3016afc36e25ea90a2d: Status 404 returned error can't find the container with id 2b2c8d3383a025d435ec2128167ef1ea721b530c9540d3016afc36e25ea90a2d
Jul 16 18:06:12.875935 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:12.875911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" event={"ID":"00afb7b6-a55b-41e1-9bf3-674f49733fd5","Type":"ContainerStarted","Data":"2b2c8d3383a025d435ec2128167ef1ea721b530c9540d3016afc36e25ea90a2d"}
Jul 16 18:06:13.881698 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:13.881617    2577 generic.go:356] "Generic (PLEG): container finished" podID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerID="498a6cdb71c84ae8f934748ba9939f3824fdd47ed0ec5387c2e826feec66d9a6" exitCode=0
Jul 16 18:06:13.882081 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:13.881708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" event={"ID":"00afb7b6-a55b-41e1-9bf3-674f49733fd5","Type":"ContainerDied","Data":"498a6cdb71c84ae8f934748ba9939f3824fdd47ed0ec5387c2e826feec66d9a6"}
Jul 16 18:06:14.886904 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:14.886864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" event={"ID":"00afb7b6-a55b-41e1-9bf3-674f49733fd5","Type":"ContainerStarted","Data":"f17b9618a2279628136374141ba7580472e4eead0d2588366362a449ec80aab9"}
Jul 16 18:06:14.887274 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:14.886975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:06:14.914919 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:14.914874    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" podStartSLOduration=2.185339322 podStartE2EDuration="2.914861409s" totalImagesPullingTime="0s" totalInitContainerRuntime="729.522087ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:06:14.913335181 +0000 UTC m=+1901.669743240" watchObservedRunningTime="2026-07-16 18:06:14.914861409 +0000 UTC m=+1901.671269471"
Jul 16 18:06:27.100494 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:27.100458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:06:45.892855 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:06:45.892825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:07:26.487899 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:07:26.487813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:10.628554 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:10.628519    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:10.652801 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:10.652774    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:08:10.692941 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:10.692914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:19.566328 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:19.566291    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:08:25.673255 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:25.673215    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerName="main" containerID="cri-o://8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7" gracePeriod=45
Jul 16 18:08:25.922108 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:25.922083    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:08:26.059903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.059875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt92f\" (UniqueName: \"kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f\") pod \"0f54d93a-7e31-4901-9981-e0ffe59cea72\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") "
Jul 16 18:08:26.060051 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.059972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds\") pod \"0f54d93a-7e31-4901-9981-e0ffe59cea72\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") "
Jul 16 18:08:26.060051 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.060010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs\") pod \"0f54d93a-7e31-4901-9981-e0ffe59cea72\" (UID: \"0f54d93a-7e31-4901-9981-e0ffe59cea72\") "
Jul 16 18:08:26.060206 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.060177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0f54d93a-7e31-4901-9981-e0ffe59cea72" (UID: "0f54d93a-7e31-4901-9981-e0ffe59cea72"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:26.061973 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.061946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f" (OuterVolumeSpecName: "kube-api-access-rt92f") pod "0f54d93a-7e31-4901-9981-e0ffe59cea72" (UID: "0f54d93a-7e31-4901-9981-e0ffe59cea72"). InnerVolumeSpecName "kube-api-access-rt92f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:08:26.062068 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.061984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f54d93a-7e31-4901-9981-e0ffe59cea72" (UID: "0f54d93a-7e31-4901-9981-e0ffe59cea72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:08:26.161510 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.161480    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0f54d93a-7e31-4901-9981-e0ffe59cea72-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:08:26.161510 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.161514    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f54d93a-7e31-4901-9981-e0ffe59cea72-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:08:26.161635 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.161523    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt92f\" (UniqueName: \"kubernetes.io/projected/0f54d93a-7e31-4901-9981-e0ffe59cea72-kube-api-access-rt92f\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:08:26.463771 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.463742    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerID="8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7" exitCode=0
Jul 16 18:08:26.463914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.463784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"
Jul 16 18:08:26.463914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.463791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" event={"ID":"0f54d93a-7e31-4901-9981-e0ffe59cea72","Type":"ContainerDied","Data":"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"}
Jul 16 18:08:26.463914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.463843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g" event={"ID":"0f54d93a-7e31-4901-9981-e0ffe59cea72","Type":"ContainerDied","Data":"4273b770ae192b00695c89e166a7555b96ee6959cf77d818440ea8b53bba36ea"}
Jul 16 18:08:26.463914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.463861    2577 scope.go:117] "RemoveContainer" containerID="8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"
Jul 16 18:08:26.471686 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.471670    2577 scope.go:117] "RemoveContainer" containerID="8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"
Jul 16 18:08:26.471958 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:08:26.471938    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7\": container with ID starting with 8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7 not found: ID does not exist" containerID="8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"
Jul 16 18:08:26.472035 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.471966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7"} err="failed to get container status \"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7\": rpc error: code = NotFound desc = could not find container \"8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7\": container with ID starting with 8584e809472f24b0c5db5f0546fe261880a6bd90d9ee1558ee447e644d4dd1f7 not found: ID does not exist"
Jul 16 18:08:26.486809 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.486787    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:08:26.490673 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:26.490653    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schetkz8g"]
Jul 16 18:08:28.304429 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:28.304396    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" path="/var/lib/kubelet/pods/0f54d93a-7e31-4901-9981-e0ffe59cea72/volumes"
Jul 16 18:08:33.145988 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.145957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:33.247288 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.247257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:33.362552 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.362521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:33.411547 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.411452    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:08:33.411870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.411848    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerName="main"
Jul 16 18:08:33.411870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.411870    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerName="main"
Jul 16 18:08:33.412050 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.412010    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f54d93a-7e31-4901-9981-e0ffe59cea72" containerName="main"
Jul 16 18:08:33.416366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.416347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.419452 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.419428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-76v2q\""
Jul 16 18:08:33.420476 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.420462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 16 18:08:33.441002 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.440978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:08:33.476953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.476928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:33.518795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.518765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.518957 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.518893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.518957 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.518949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxwg\" (UniqueName: \"kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.620228 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.620191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.620228 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.620228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxwg\" (UniqueName: \"kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.620459 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.620362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.620765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.620747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.622860 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.622841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.640274 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.640247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxwg\" (UniqueName: \"kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.726460 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.726402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:33.853288 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:33.853264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:08:33.854785 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:08:33.854760    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600fe86c_c54e_42d5_b53d_00395a971538.slice/crio-edb6bfa8a6e6c2e1365e17d4ef98e0c38a057d8625d4feaea9d57c23fdcf839c WatchSource:0}: Error finding container edb6bfa8a6e6c2e1365e17d4ef98e0c38a057d8625d4feaea9d57c23fdcf839c: Status 404 returned error can't find the container with id edb6bfa8a6e6c2e1365e17d4ef98e0c38a057d8625d4feaea9d57c23fdcf839c
Jul 16 18:08:34.498567 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:34.498533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" event={"ID":"600fe86c-c54e-42d5-b53d-00395a971538","Type":"ContainerStarted","Data":"ccbbe6c2ba33767faa8e29f1f32a38683098c4f83bea7ffcb0a024c0a6eec928"}
Jul 16 18:08:34.498965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:34.498609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" event={"ID":"600fe86c-c54e-42d5-b53d-00395a971538","Type":"ContainerStarted","Data":"edb6bfa8a6e6c2e1365e17d4ef98e0c38a057d8625d4feaea9d57c23fdcf839c"}
Jul 16 18:08:34.498965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:34.498673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:08:34.517888 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:34.517840    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" podStartSLOduration=1.517819189 podStartE2EDuration="1.517819189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:34.517287705 +0000 UTC m=+2041.273695769" watchObservedRunningTime="2026-07-16 18:08:34.517819189 +0000 UTC m=+2041.274227281"
Jul 16 18:08:39.535469 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:39.535425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:51.437046 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:51.437013    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:08:51.444780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:51.444754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:51.500607 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:51.500578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:08:55.892762 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:08:55.892677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:09:05.504736 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:05.504704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:09:05.892823 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:05.892742    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:09:06.457215 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.457171    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main" containerID="cri-o://f17b9618a2279628136374141ba7580472e4eead0d2588366362a449ec80aab9" gracePeriod=45
Jul 16 18:09:06.638359 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.638329    2577 generic.go:356] "Generic (PLEG): container finished" podID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerID="f17b9618a2279628136374141ba7580472e4eead0d2588366362a449ec80aab9" exitCode=0
Jul 16 18:09:06.638691 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.638406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" event={"ID":"00afb7b6-a55b-41e1-9bf3-674f49733fd5","Type":"ContainerDied","Data":"f17b9618a2279628136374141ba7580472e4eead0d2588366362a449ec80aab9"}
Jul 16 18:09:06.710988 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.710934    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:09:06.787620 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.787591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location\") pod \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") "
Jul 16 18:09:06.787753 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.787646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds\") pod \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") "
Jul 16 18:09:06.787753 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.787691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs\") pod \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") "
Jul 16 18:09:06.787753 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.787723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgqw9\" (UniqueName: \"kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9\") pod \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\" (UID: \"00afb7b6-a55b-41e1-9bf3-674f49733fd5\") "
Jul 16 18:09:06.787950 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.787926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "00afb7b6-a55b-41e1-9bf3-674f49733fd5" (UID: "00afb7b6-a55b-41e1-9bf3-674f49733fd5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:06.788306 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.788284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00afb7b6-a55b-41e1-9bf3-674f49733fd5" (UID: "00afb7b6-a55b-41e1-9bf3-674f49733fd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:06.789834 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.789815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00afb7b6-a55b-41e1-9bf3-674f49733fd5" (UID: "00afb7b6-a55b-41e1-9bf3-674f49733fd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:09:06.789834 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.789820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9" (OuterVolumeSpecName: "kube-api-access-mgqw9") pod "00afb7b6-a55b-41e1-9bf3-674f49733fd5" (UID: "00afb7b6-a55b-41e1-9bf3-674f49733fd5"). InnerVolumeSpecName "kube-api-access-mgqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:09:06.888657 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.888635    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:06.888657 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.888654    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:06.888776 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.888664    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00afb7b6-a55b-41e1-9bf3-674f49733fd5-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:06.888776 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:06.888674    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqw9\" (UniqueName: \"kubernetes.io/projected/00afb7b6-a55b-41e1-9bf3-674f49733fd5-kube-api-access-mgqw9\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.643827 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.643786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk" event={"ID":"00afb7b6-a55b-41e1-9bf3-674f49733fd5","Type":"ContainerDied","Data":"2b2c8d3383a025d435ec2128167ef1ea721b530c9540d3016afc36e25ea90a2d"}
Jul 16 18:09:07.643827 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.643806    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"
Jul 16 18:09:07.644356 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.643858    2577 scope.go:117] "RemoveContainer" containerID="f17b9618a2279628136374141ba7580472e4eead0d2588366362a449ec80aab9"
Jul 16 18:09:07.652371 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.652355    2577 scope.go:117] "RemoveContainer" containerID="498a6cdb71c84ae8f934748ba9939f3824fdd47ed0ec5387c2e826feec66d9a6"
Jul 16 18:09:07.677670 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.677638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:09:07.681721 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:07.681700    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6869bff88sqkk"]
Jul 16 18:09:08.304050 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:08.304009    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" path="/var/lib/kubelet/pods/00afb7b6-a55b-41e1-9bf3-674f49733fd5/volumes"
Jul 16 18:09:14.409837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:14.409801    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 18:09:14.410238 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:14.410023    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy" containerID="cri-o://fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c" gracePeriod=30
Jul 16 18:09:19.658407 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.658381    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 18:09:19.684286 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684286 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvvx8\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684526 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert\") pod \"123399d5-de98-4560-b7e3-37814a765e9b\" (UID: \"123399d5-de98-4560-b7e3-37814a765e9b\") "
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684747    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684852    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684855    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684872    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.684886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.684887    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.685241 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.685071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:19.687073 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.687017    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:19.687073 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.687025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:09:19.687247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.687118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8" (OuterVolumeSpecName: "kube-api-access-qvvx8") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "kube-api-access-qvvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:09:19.687804 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.687779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "123399d5-de98-4560-b7e3-37814a765e9b" (UID: "123399d5-de98-4560-b7e3-37814a765e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 18:09:19.696732 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.696708    2577 generic.go:356] "Generic (PLEG): container finished" podID="123399d5-de98-4560-b7e3-37814a765e9b" containerID="fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c" exitCode=0
Jul 16 18:09:19.696835 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.696763    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"
Jul 16 18:09:19.696835 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.696786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerDied","Data":"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"}
Jul 16 18:09:19.696835 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.696834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh" event={"ID":"123399d5-de98-4560-b7e3-37814a765e9b","Type":"ContainerDied","Data":"6d85d8b42b8829907697611bcf568061284a265a1d98c9754e4232bae0129316"}
Jul 16 18:09:19.696998 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.696855    2577 scope.go:117] "RemoveContainer" containerID="fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"
Jul 16 18:09:19.706154 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.706085    2577 scope.go:117] "RemoveContainer" containerID="d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"
Jul 16 18:09:19.713279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.713260    2577 scope.go:117] "RemoveContainer" containerID="fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"
Jul 16 18:09:19.713618 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:09:19.713598    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c\": container with ID starting with fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c not found: ID does not exist" containerID="fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"
Jul 16 18:09:19.713663 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.713627    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c"} err="failed to get container status \"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c\": rpc error: code = NotFound desc = could not find container \"fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c\": container with ID starting with fdfc9b4e8e10c6df41efe3f80ac16873f4225af0114e7b9a1f6afa9204a51e7c not found: ID does not exist"
Jul 16 18:09:19.713663 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.713643    2577 scope.go:117] "RemoveContainer" containerID="d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"
Jul 16 18:09:19.713880 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:09:19.713860    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a\": container with ID starting with d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a not found: ID does not exist" containerID="d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"
Jul 16 18:09:19.713943 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.713889    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a"} err="failed to get container status \"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a\": rpc error: code = NotFound desc = could not find container \"d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a\": container with ID starting with d06ad16173e1a92655033082e5722d6e6bad58391c34fd7360ff861d5a2a6d1a not found: ID does not exist"
Jul 16 18:09:19.730681 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.730659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 18:09:19.736714 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.736692    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-p4vzh"]
Jul 16 18:09:19.785954 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785933    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.785954 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785953    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvx8\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-kube-api-access-qvvx8\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.786080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785962    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123399d5-de98-4560-b7e3-37814a765e9b-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.786080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785972    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123399d5-de98-4560-b7e3-37814a765e9b-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.786080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785980    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123399d5-de98-4560-b7e3-37814a765e9b-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:19.786080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:19.785987    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123399d5-de98-4560-b7e3-37814a765e9b-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:09:20.304210 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:20.304170    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123399d5-de98-4560-b7e3-37814a765e9b" path="/var/lib/kubelet/pods/123399d5-de98-4560-b7e3-37814a765e9b/volumes"
Jul 16 18:09:32.236792 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.236756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:09:32.988884 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.988851    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:09:32.989136 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989125    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.989182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989138    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.989182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989152    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main"
Jul 16 18:09:32.989182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989157    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main"
Jul 16 18:09:32.989182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989165    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.989182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989171    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.989332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989186    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="storage-initializer"
Jul 16 18:09:32.989332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989192    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="storage-initializer"
Jul 16 18:09:32.989332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989235    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.989332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989243    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="00afb7b6-a55b-41e1-9bf3-674f49733fd5" containerName="main"
Jul 16 18:09:32.989332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.989330    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="123399d5-de98-4560-b7e3-37814a765e9b" containerName="istio-proxy"
Jul 16 18:09:32.993439 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:32.993423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.007322 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.007305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:09:33.008288 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.008273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-6ctzc\""
Jul 16 18:09:33.008288 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.008284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 16 18:09:33.020055 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.020023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:09:33.028911 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.028857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:09:33.081129 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.081104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.081232 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.081136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.081232 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.081165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.081312 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.081266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45cc\" (UniqueName: \"kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.118302 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.118276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:09:33.182282 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.182392 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.182392 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.182392 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45cc\" (UniqueName: \"kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.182812 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.182876 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.182837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.184789 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.184770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.192361 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.192339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45cc\" (UniqueName: \"kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc\") pod \"router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.301616 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.301592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:33.435297 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.435162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:09:33.437918 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:09:33.437890    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b5b62d_d869_41c8_9b64_9ebac18552e2.slice/crio-0415edc87fe8f98f9b372090f517030280fa51d67b39d61c1ee84f37a916e405 WatchSource:0}: Error finding container 0415edc87fe8f98f9b372090f517030280fa51d67b39d61c1ee84f37a916e405: Status 404 returned error can't find the container with id 0415edc87fe8f98f9b372090f517030280fa51d67b39d61c1ee84f37a916e405
Jul 16 18:09:33.760765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.760728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerStarted","Data":"6adb0549d2f3b6c96a13b4cb16b21c4cf0c1ad253c0040f03d61970362748257"}
Jul 16 18:09:33.760937 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:33.760782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerStarted","Data":"0415edc87fe8f98f9b372090f517030280fa51d67b39d61c1ee84f37a916e405"}
Jul 16 18:09:34.765885 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:34.765854    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerID="6adb0549d2f3b6c96a13b4cb16b21c4cf0c1ad253c0040f03d61970362748257" exitCode=0
Jul 16 18:09:34.766330 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:34.765914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerDied","Data":"6adb0549d2f3b6c96a13b4cb16b21c4cf0c1ad253c0040f03d61970362748257"}
Jul 16 18:09:35.771604 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:35.771574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerStarted","Data":"55ac18e662d89e47f2c5bd11b1abfd8fc94f4a5119b042b2861bc56e7eb5d208"}
Jul 16 18:09:35.771998 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:35.771684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:09:35.796305 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:09:35.796260    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" podStartSLOduration=3.008558131 podStartE2EDuration="3.796247833s" totalImagesPullingTime="0s" totalInitContainerRuntime="787.689702ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:09:35.794650708 +0000 UTC m=+2102.551058776" watchObservedRunningTime="2026-07-16 18:09:35.796247833 +0000 UTC m=+2102.552655899"
Jul 16 18:10:06.777691 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:10:06.777660    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:11:47.094353 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:11:47.094322    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:11:47.096563 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:11:47.096537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:11:48.097186 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:11:48.097153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:11:56.778018 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:11:56.777938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:12:02.114171 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.114129    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="main" containerID="cri-o://55ac18e662d89e47f2c5bd11b1abfd8fc94f4a5119b042b2861bc56e7eb5d208" gracePeriod=45
Jul 16 18:12:02.345703 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.345662    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerID="55ac18e662d89e47f2c5bd11b1abfd8fc94f4a5119b042b2861bc56e7eb5d208" exitCode=0
Jul 16 18:12:02.345835 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.345727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerDied","Data":"55ac18e662d89e47f2c5bd11b1abfd8fc94f4a5119b042b2861bc56e7eb5d208"}
Jul 16 18:12:02.360419 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.360393    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:12:02.405186 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q45cc\" (UniqueName: \"kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc\") pod \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") "
Jul 16 18:12:02.405298 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location\") pod \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") "
Jul 16 18:12:02.405298 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds\") pod \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") "
Jul 16 18:12:02.405428 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs\") pod \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\" (UID: \"e8b5b62d-d869-41c8-9b64-9ebac18552e2\") "
Jul 16 18:12:02.405615 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e8b5b62d-d869-41c8-9b64-9ebac18552e2" (UID: "e8b5b62d-d869-41c8-9b64-9ebac18552e2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:02.405939 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.405910    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8b5b62d-d869-41c8-9b64-9ebac18552e2" (UID: "e8b5b62d-d869-41c8-9b64-9ebac18552e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:02.407055 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.407035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc" (OuterVolumeSpecName: "kube-api-access-q45cc") pod "e8b5b62d-d869-41c8-9b64-9ebac18552e2" (UID: "e8b5b62d-d869-41c8-9b64-9ebac18552e2"). InnerVolumeSpecName "kube-api-access-q45cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:12:02.407467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.407442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8b5b62d-d869-41c8-9b64-9ebac18552e2" (UID: "e8b5b62d-d869-41c8-9b64-9ebac18552e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:12:02.506733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.506710    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:02.506733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.506730    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:02.506858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.506739    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b5b62d-d869-41c8-9b64-9ebac18552e2-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:02.506858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:02.506747    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q45cc\" (UniqueName: \"kubernetes.io/projected/e8b5b62d-d869-41c8-9b64-9ebac18552e2-kube-api-access-q45cc\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.351098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.351067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg" event={"ID":"e8b5b62d-d869-41c8-9b64-9ebac18552e2","Type":"ContainerDied","Data":"0415edc87fe8f98f9b372090f517030280fa51d67b39d61c1ee84f37a916e405"}
Jul 16 18:12:03.351098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.351097    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"
Jul 16 18:12:03.351578 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.351120    2577 scope.go:117] "RemoveContainer" containerID="55ac18e662d89e47f2c5bd11b1abfd8fc94f4a5119b042b2861bc56e7eb5d208"
Jul 16 18:12:03.359801 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.359786    2577 scope.go:117] "RemoveContainer" containerID="6adb0549d2f3b6c96a13b4cb16b21c4cf0c1ad253c0040f03d61970362748257"
Jul 16 18:12:03.376201 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.376180    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:12:03.381585 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:03.381565    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-859f948bc7-vnnjg"]
Jul 16 18:12:04.304299 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:04.304258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" path="/var/lib/kubelet/pods/e8b5b62d-d869-41c8-9b64-9ebac18552e2/volumes"
Jul 16 18:12:15.021163 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.021127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:12:15.425834 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.425765    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:12:15.426035 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.426024    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="main"
Jul 16 18:12:15.426078 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.426036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="main"
Jul 16 18:12:15.426078 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.426048    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="storage-initializer"
Jul 16 18:12:15.426078 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.426054    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="storage-initializer"
Jul 16 18:12:15.426168 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.426106    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b5b62d-d869-41c8-9b64-9ebac18552e2" containerName="main"
Jul 16 18:12:15.429087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.429067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.431673 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.431648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:12:15.431865 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.431851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-hlp96\""
Jul 16 18:12:15.432161 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.432145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 16 18:12:15.441742 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.441713    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:12:15.460066 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.460043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:12:15.509529 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.509489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.509629 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.509560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.509629 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.509615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.509704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.509635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtcq\" (UniqueName: \"kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610230 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610327 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610327 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtcq\" (UniqueName: \"kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610415 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610715 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.610715 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.610709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.612662 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.612640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.620494 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.620471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtcq\" (UniqueName: \"kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.741657 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.741604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:15.878538 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:15.878493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:12:15.881127 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:12:15.881100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b20925_3545_4c80_a611_507a1141bc82.slice/crio-798c8c641378731e7aef1c3a53d8946c6a529b037536f6f75acf257f9acadd67 WatchSource:0}: Error finding container 798c8c641378731e7aef1c3a53d8946c6a529b037536f6f75acf257f9acadd67: Status 404 returned error can't find the container with id 798c8c641378731e7aef1c3a53d8946c6a529b037536f6f75acf257f9acadd67
Jul 16 18:12:16.412369 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:16.412329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerStarted","Data":"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"}
Jul 16 18:12:16.412754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:16.412397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerStarted","Data":"798c8c641378731e7aef1c3a53d8946c6a529b037536f6f75acf257f9acadd67"}
Jul 16 18:12:17.418072 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:17.418034    2577 generic.go:356] "Generic (PLEG): container finished" podID="79b20925-3545-4c80-a611-507a1141bc82" containerID="794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489" exitCode=0
Jul 16 18:12:17.418473 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:17.418120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerDied","Data":"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"}
Jul 16 18:12:18.423650 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:18.423618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerStarted","Data":"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"}
Jul 16 18:12:18.424121 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:18.423720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:12:18.447956 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:18.447906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" podStartSLOduration=2.692254597 podStartE2EDuration="3.447888098s" totalImagesPullingTime="0s" totalInitContainerRuntime="755.633501ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:12:18.446409171 +0000 UTC m=+2265.202817257" watchObservedRunningTime="2026-07-16 18:12:18.447888098 +0000 UTC m=+2265.204296163"
Jul 16 18:12:29.060014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:29.059982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:12:31.355754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:31.355720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:12:31.384357 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:31.384326    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:12:31.420981 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:31.420949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:12:35.504363 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:35.504326    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:12:45.505086 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:45.505044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:12:46.403797 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.403762    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main" containerID="cri-o://ccbbe6c2ba33767faa8e29f1f32a38683098c4f83bea7ffcb0a024c0a6eec928" gracePeriod=45
Jul 16 18:12:46.534020 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.533995    2577 generic.go:356] "Generic (PLEG): container finished" podID="600fe86c-c54e-42d5-b53d-00395a971538" containerID="ccbbe6c2ba33767faa8e29f1f32a38683098c4f83bea7ffcb0a024c0a6eec928" exitCode=0
Jul 16 18:12:46.534304 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.534060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" event={"ID":"600fe86c-c54e-42d5-b53d-00395a971538","Type":"ContainerDied","Data":"ccbbe6c2ba33767faa8e29f1f32a38683098c4f83bea7ffcb0a024c0a6eec928"}
Jul 16 18:12:46.651314 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.651294    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:12:46.752652 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.752629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds\") pod \"600fe86c-c54e-42d5-b53d-00395a971538\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") "
Jul 16 18:12:46.752754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.752709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzxwg\" (UniqueName: \"kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg\") pod \"600fe86c-c54e-42d5-b53d-00395a971538\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") "
Jul 16 18:12:46.752754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.752730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs\") pod \"600fe86c-c54e-42d5-b53d-00395a971538\" (UID: \"600fe86c-c54e-42d5-b53d-00395a971538\") "
Jul 16 18:12:46.752891 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.752863    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "600fe86c-c54e-42d5-b53d-00395a971538" (UID: "600fe86c-c54e-42d5-b53d-00395a971538"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:46.754672 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.754643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg" (OuterVolumeSpecName: "kube-api-access-zzxwg") pod "600fe86c-c54e-42d5-b53d-00395a971538" (UID: "600fe86c-c54e-42d5-b53d-00395a971538"). InnerVolumeSpecName "kube-api-access-zzxwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:12:46.754778 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.754755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "600fe86c-c54e-42d5-b53d-00395a971538" (UID: "600fe86c-c54e-42d5-b53d-00395a971538"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:12:46.853220 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.853197    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxwg\" (UniqueName: \"kubernetes.io/projected/600fe86c-c54e-42d5-b53d-00395a971538-kube-api-access-zzxwg\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:46.853220 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.853216    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600fe86c-c54e-42d5-b53d-00395a971538-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:46.853352 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:46.853227    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/600fe86c-c54e-42d5-b53d-00395a971538-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.540162 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:47.540127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4" event={"ID":"600fe86c-c54e-42d5-b53d-00395a971538","Type":"ContainerDied","Data":"edb6bfa8a6e6c2e1365e17d4ef98e0c38a057d8625d4feaea9d57c23fdcf839c"}
Jul 16 18:12:47.540162 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:47.540153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"
Jul 16 18:12:47.540645 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:47.540207    2577 scope.go:117] "RemoveContainer" containerID="ccbbe6c2ba33767faa8e29f1f32a38683098c4f83bea7ffcb0a024c0a6eec928"
Jul 16 18:12:47.564383 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:47.564359    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:12:47.569429 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:47.569407    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schepj8r4"]
Jul 16 18:12:48.304080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:48.304050    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600fe86c-c54e-42d5-b53d-00395a971538" path="/var/lib/kubelet/pods/600fe86c-c54e-42d5-b53d-00395a971538/volumes"
Jul 16 18:12:49.429132 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:12:49.429104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:13:11.872933 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:11.872901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:13:12.131283 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.131207    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:13:12.131553 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.131540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main"
Jul 16 18:13:12.131605 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.131555    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main"
Jul 16 18:13:12.131641 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.131608    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="600fe86c-c54e-42d5-b53d-00395a971538" containerName="main"
Jul 16 18:13:12.134498 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.134483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.137119 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.137098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-8hpdf\""
Jul 16 18:13:12.137520 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.137486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 18:13:12.151368 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.151344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:13:12.161950 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.161927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:13:12.243577 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.243545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.243719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.243598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.243719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.243657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.243719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.243686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n84s\" (UniqueName: \"kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.344863 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.344832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.344987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.344881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.344987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.344916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.344987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.344943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n84s\" (UniqueName: \"kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.345236 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.345214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.345303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.345239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.347295 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.347276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.354924 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.354898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n84s\" (UniqueName: \"kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-jljrz\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.443822 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.443769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:12.587418 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.587395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:13:12.591192 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:13:12.591158    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb536203c_7096_4d21_b5c4_62310e63911e.slice/crio-aef7547fa8f1e72be0d0d079c05d112f38174e56c82cc89b28e33e80f1cbfb65 WatchSource:0}: Error finding container aef7547fa8f1e72be0d0d079c05d112f38174e56c82cc89b28e33e80f1cbfb65: Status 404 returned error can't find the container with id aef7547fa8f1e72be0d0d079c05d112f38174e56c82cc89b28e33e80f1cbfb65
Jul 16 18:13:12.643257 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:12.643233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" event={"ID":"b536203c-7096-4d21-b5c4-62310e63911e","Type":"ContainerStarted","Data":"aef7547fa8f1e72be0d0d079c05d112f38174e56c82cc89b28e33e80f1cbfb65"}
Jul 16 18:13:13.648450 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:13.648416    2577 generic.go:356] "Generic (PLEG): container finished" podID="b536203c-7096-4d21-b5c4-62310e63911e" containerID="865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca" exitCode=0
Jul 16 18:13:13.648887 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:13.648482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" event={"ID":"b536203c-7096-4d21-b5c4-62310e63911e","Type":"ContainerDied","Data":"865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca"}
Jul 16 18:13:14.654361 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:14.654322    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" event={"ID":"b536203c-7096-4d21-b5c4-62310e63911e","Type":"ContainerStarted","Data":"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"}
Jul 16 18:13:14.654739 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:14.654416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:13:14.699355 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:14.699299    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" podStartSLOduration=1.937730602 podStartE2EDuration="2.699284421s" totalImagesPullingTime="0s" totalInitContainerRuntime="761.553819ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:13:14.698602132 +0000 UTC m=+2321.455010194" watchObservedRunningTime="2026-07-16 18:13:14.699284421 +0000 UTC m=+2321.455692483"
Jul 16 18:13:36.760893 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:36.760861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:13:45.660921 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:13:45.660887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:14:30.035367 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:14:30.035327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:14:30.134690 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:14:30.134656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:01.304130 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:01.304043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:17.688236 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:17.688199    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:34.857552 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:34.857516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:40.532318 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:40.532275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:40.575630 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:40.575592    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:15:40.617560 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:40.617530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:49.428783 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:49.428744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:15:53.794113 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:53.794072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:15:55.597150 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:55.597088    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="main" containerID="cri-o://82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d" gracePeriod=45
Jul 16 18:15:55.842884 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:55.842863    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:15:56.006588 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.006565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs\") pod \"79b20925-3545-4c80-a611-507a1141bc82\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") "
Jul 16 18:15:56.006704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.006603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location\") pod \"79b20925-3545-4c80-a611-507a1141bc82\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") "
Jul 16 18:15:56.006704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.006661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhtcq\" (UniqueName: \"kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq\") pod \"79b20925-3545-4c80-a611-507a1141bc82\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") "
Jul 16 18:15:56.006704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.006680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds\") pod \"79b20925-3545-4c80-a611-507a1141bc82\" (UID: \"79b20925-3545-4c80-a611-507a1141bc82\") "
Jul 16 18:15:56.007001 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.006975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "79b20925-3545-4c80-a611-507a1141bc82" (UID: "79b20925-3545-4c80-a611-507a1141bc82"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.007314 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.007293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79b20925-3545-4c80-a611-507a1141bc82" (UID: "79b20925-3545-4c80-a611-507a1141bc82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.008645 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.008617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq" (OuterVolumeSpecName: "kube-api-access-nhtcq") pod "79b20925-3545-4c80-a611-507a1141bc82" (UID: "79b20925-3545-4c80-a611-507a1141bc82"). InnerVolumeSpecName "kube-api-access-nhtcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:15:56.008757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.008641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79b20925-3545-4c80-a611-507a1141bc82" (UID: "79b20925-3545-4c80-a611-507a1141bc82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:15:56.107294 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.107258    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b20925-3545-4c80-a611-507a1141bc82-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.107294 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.107293    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.107425 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.107307    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtcq\" (UniqueName: \"kubernetes.io/projected/79b20925-3545-4c80-a611-507a1141bc82-kube-api-access-nhtcq\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.107425 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.107321    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/79b20925-3545-4c80-a611-507a1141bc82-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.320127 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.320101    2577 generic.go:356] "Generic (PLEG): container finished" podID="79b20925-3545-4c80-a611-507a1141bc82" containerID="82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d" exitCode=0
Jul 16 18:15:56.320247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.320147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"
Jul 16 18:15:56.320247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.320181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerDied","Data":"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"}
Jul 16 18:15:56.320247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.320235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6" event={"ID":"79b20925-3545-4c80-a611-507a1141bc82","Type":"ContainerDied","Data":"798c8c641378731e7aef1c3a53d8946c6a529b037536f6f75acf257f9acadd67"}
Jul 16 18:15:56.320416 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.320262    2577 scope.go:117] "RemoveContainer" containerID="82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"
Jul 16 18:15:56.328270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.328255    2577 scope.go:117] "RemoveContainer" containerID="794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"
Jul 16 18:15:56.335241 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.335225    2577 scope.go:117] "RemoveContainer" containerID="82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"
Jul 16 18:15:56.335511 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:15:56.335482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d\": container with ID starting with 82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d not found: ID does not exist" containerID="82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"
Jul 16 18:15:56.335576 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.335531    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d"} err="failed to get container status \"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d\": rpc error: code = NotFound desc = could not find container \"82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d\": container with ID starting with 82ba70620191b2cd39cf696babea37d1b263dae98a7c30c95554491198d1187d not found: ID does not exist"
Jul 16 18:15:56.335576 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.335554    2577 scope.go:117] "RemoveContainer" containerID="794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"
Jul 16 18:15:56.335829 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:15:56.335805    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489\": container with ID starting with 794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489 not found: ID does not exist" containerID="794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"
Jul 16 18:15:56.335872 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.335837    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489"} err="failed to get container status \"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489\": rpc error: code = NotFound desc = could not find container \"794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489\": container with ID starting with 794240938a0ef5c00cfff65e3b2a0889f08fe41a6a791e5611681a1665f89489 not found: ID does not exist"
Jul 16 18:15:56.340072 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.340051    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:15:56.344608 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:56.344589    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche68br6"]
Jul 16 18:15:58.304115 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:15:58.304080    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b20925-3545-4c80-a611-507a1141bc82" path="/var/lib/kubelet/pods/79b20925-3545-4c80-a611-507a1141bc82/volumes"
Jul 16 18:16:02.967091 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:02.967055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:16:03.074450 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:03.074408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:03.174787 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:03.174735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:05.660900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:05.660854    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:16:15.660900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:15.660862    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:16:17.832193 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:17.832124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:17.929141 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:17.929109    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:17.986356 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:17.986318    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main" containerID="cri-o://561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53" gracePeriod=45
Jul 16 18:16:18.071368 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071335    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:16:18.071661 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071648    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="main"
Jul 16 18:16:18.071712 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="main"
Jul 16 18:16:18.071712 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071673    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="storage-initializer"
Jul 16 18:16:18.071712 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071680    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="storage-initializer"
Jul 16 18:16:18.071809 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.071733    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b20925-3545-4c80-a611-507a1141bc82" containerName="main"
Jul 16 18:16:18.074797 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.074781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.077270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.077250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 16 18:16:18.077389 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.077269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-9fbxb\""
Jul 16 18:16:18.087633 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.087582    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:16:18.100095 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.100065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:18.171242 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.171212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.171415 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.171379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4sqk\" (UniqueName: \"kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.171478 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.171464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.171559 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.171539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.237907 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.237874    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:16:18.272487 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location\") pod \"b536203c-7096-4d21-b5c4-62310e63911e\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") "
Jul 16 18:16:18.272635 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds\") pod \"b536203c-7096-4d21-b5c4-62310e63911e\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") "
Jul 16 18:16:18.272635 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs\") pod \"b536203c-7096-4d21-b5c4-62310e63911e\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") "
Jul 16 18:16:18.272635 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n84s\" (UniqueName: \"kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s\") pod \"b536203c-7096-4d21-b5c4-62310e63911e\" (UID: \"b536203c-7096-4d21-b5c4-62310e63911e\") "
Jul 16 18:16:18.272806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b536203c-7096-4d21-b5c4-62310e63911e" (UID: "b536203c-7096-4d21-b5c4-62310e63911e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:18.272806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.272914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4sqk\" (UniqueName: \"kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.272983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.272983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.272966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.273090 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.273069    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:16:18.273441 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.273418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.273572 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.273473    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b536203c-7096-4d21-b5c4-62310e63911e" (UID: "b536203c-7096-4d21-b5c4-62310e63911e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:18.273646 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.273582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.274713 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.274684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b536203c-7096-4d21-b5c4-62310e63911e" (UID: "b536203c-7096-4d21-b5c4-62310e63911e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:16:18.274799 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.274773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s" (OuterVolumeSpecName: "kube-api-access-7n84s") pod "b536203c-7096-4d21-b5c4-62310e63911e" (UID: "b536203c-7096-4d21-b5c4-62310e63911e"). InnerVolumeSpecName "kube-api-access-7n84s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:16:18.275445 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.275428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.282091 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.282067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4sqk\" (UniqueName: \"kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.374074 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.374005    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b536203c-7096-4d21-b5c4-62310e63911e-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:16:18.374074 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.374025    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b536203c-7096-4d21-b5c4-62310e63911e-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:16:18.374074 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.374036    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n84s\" (UniqueName: \"kubernetes.io/projected/b536203c-7096-4d21-b5c4-62310e63911e-kube-api-access-7n84s\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:16:18.383974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.383941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:18.406639 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.406604    2577 generic.go:356] "Generic (PLEG): container finished" podID="b536203c-7096-4d21-b5c4-62310e63911e" containerID="561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53" exitCode=0
Jul 16 18:16:18.406748 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.406712    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"
Jul 16 18:16:18.406806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.406706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" event={"ID":"b536203c-7096-4d21-b5c4-62310e63911e","Type":"ContainerDied","Data":"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"}
Jul 16 18:16:18.406864 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.406839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz" event={"ID":"b536203c-7096-4d21-b5c4-62310e63911e","Type":"ContainerDied","Data":"aef7547fa8f1e72be0d0d079c05d112f38174e56c82cc89b28e33e80f1cbfb65"}
Jul 16 18:16:18.406922 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.406871    2577 scope.go:117] "RemoveContainer" containerID="561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"
Jul 16 18:16:18.416023 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.416010    2577 scope.go:117] "RemoveContainer" containerID="865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca"
Jul 16 18:16:18.423301 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.423280    2577 scope.go:117] "RemoveContainer" containerID="561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"
Jul 16 18:16:18.424011 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:16:18.423990    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53\": container with ID starting with 561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53 not found: ID does not exist" containerID="561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"
Jul 16 18:16:18.424092 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.424019    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53"} err="failed to get container status \"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53\": rpc error: code = NotFound desc = could not find container \"561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53\": container with ID starting with 561d88ec1a00da1fb6d9dea58db98be3e259d22640a37349fbe561b1d478af53 not found: ID does not exist"
Jul 16 18:16:18.424092 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.424038    2577 scope.go:117] "RemoveContainer" containerID="865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca"
Jul 16 18:16:18.424305 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:16:18.424285    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca\": container with ID starting with 865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca not found: ID does not exist" containerID="865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca"
Jul 16 18:16:18.424399 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.424313    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca"} err="failed to get container status \"865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca\": rpc error: code = NotFound desc = could not find container \"865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca\": container with ID starting with 865f9bf5d35cf39fbd2ab8fb7c0028b7870f86e436ff924f51e2c25049f898ca not found: ID does not exist"
Jul 16 18:16:18.424622 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.424494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:16:18.427826 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.427798    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-jljrz"]
Jul 16 18:16:18.515288 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:18.515145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:16:18.517565 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:16:18.517538    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9818ffb_91f0_47b0_ae3e_1ded1f490e26.slice/crio-b7206ae68754c3d6b91f3190b48dc36e17de2ef79d6871ad63c12ca7b956f938 WatchSource:0}: Error finding container b7206ae68754c3d6b91f3190b48dc36e17de2ef79d6871ad63c12ca7b956f938: Status 404 returned error can't find the container with id b7206ae68754c3d6b91f3190b48dc36e17de2ef79d6871ad63c12ca7b956f938
Jul 16 18:16:19.411926 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:19.411892    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerID="c6d03e0045bfb7aaf07eccf7e6d9f235a4737c7caef64945dd6db58375e2e454" exitCode=0
Jul 16 18:16:19.412342 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:19.411981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" event={"ID":"f9818ffb-91f0-47b0-ae3e-1ded1f490e26","Type":"ContainerDied","Data":"c6d03e0045bfb7aaf07eccf7e6d9f235a4737c7caef64945dd6db58375e2e454"}
Jul 16 18:16:19.412342 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:19.412073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" event={"ID":"f9818ffb-91f0-47b0-ae3e-1ded1f490e26","Type":"ContainerStarted","Data":"b7206ae68754c3d6b91f3190b48dc36e17de2ef79d6871ad63c12ca7b956f938"}
Jul 16 18:16:20.305089 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:20.305056    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b536203c-7096-4d21-b5c4-62310e63911e" path="/var/lib/kubelet/pods/b536203c-7096-4d21-b5c4-62310e63911e/volumes"
Jul 16 18:16:20.418364 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:20.418332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" event={"ID":"f9818ffb-91f0-47b0-ae3e-1ded1f490e26","Type":"ContainerStarted","Data":"7fbab6903b1a1e4bec9ca555e58bdf7bbfce73d69c8beb656d574a137eb53a00"}
Jul 16 18:16:20.418754 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:20.418424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:16:20.441722 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:20.441663    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" podStartSLOduration=1.719302728 podStartE2EDuration="2.44164274s" totalImagesPullingTime="0s" totalInitContainerRuntime="722.340012ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:16:20.438985102 +0000 UTC m=+2507.195393165" watchObservedRunningTime="2026-07-16 18:16:20.44164274 +0000 UTC m=+2507.198050803"
Jul 16 18:16:28.432680 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.432605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:28.732239 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732159    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:16:28.732481 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732468    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main"
Jul 16 18:16:28.732551 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732482    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main"
Jul 16 18:16:28.732551 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="storage-initializer"
Jul 16 18:16:28.732551 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732532    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="storage-initializer"
Jul 16 18:16:28.732653 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.732583    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b536203c-7096-4d21-b5c4-62310e63911e" containerName="main"
Jul 16 18:16:28.736720 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.736702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.739323 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.739300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-mdqxx\""
Jul 16 18:16:28.740204 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.740186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 18:16:28.747613 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.747591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:16:28.750231 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.750208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.750333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.750256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.750333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.750280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvzv\" (UniqueName: \"kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.750333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.750319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.759471 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.759449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:28.851260 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.851362 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.851362 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvzv\" (UniqueName: \"kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.851362 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.851701 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.851749 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.851732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.853794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.853776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:28.861190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:28.861170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvzv\" (UniqueName: \"kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv\") pod \"stop-feature-test-kserve-router-scheduler-546967d655-nrcg7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:29.046241 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:29.046215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:29.205050 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:29.205019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:16:29.207993 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:16:29.207968    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012fda12_abfb_4048_a270_ded28ced8ec7.slice/crio-5eabc438c201ce48f321420665a429e861af7096fc50949f2ac80b4170fb7de7 WatchSource:0}: Error finding container 5eabc438c201ce48f321420665a429e861af7096fc50949f2ac80b4170fb7de7: Status 404 returned error can't find the container with id 5eabc438c201ce48f321420665a429e861af7096fc50949f2ac80b4170fb7de7
Jul 16 18:16:29.455965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:29.455871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerStarted","Data":"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"}
Jul 16 18:16:29.455965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:29.455935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerStarted","Data":"5eabc438c201ce48f321420665a429e861af7096fc50949f2ac80b4170fb7de7"}
Jul 16 18:16:30.461805 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:30.461769    2577 generic.go:356] "Generic (PLEG): container finished" podID="012fda12-abfb-4048-a270-ded28ced8ec7" containerID="e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769" exitCode=0
Jul 16 18:16:30.462162 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:30.461860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerDied","Data":"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"}
Jul 16 18:16:31.468494 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:31.468457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerStarted","Data":"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"}
Jul 16 18:16:31.468912 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:31.468621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:16:31.488232 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:31.488180    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" podStartSLOduration=2.752481488 podStartE2EDuration="3.488161967s" totalImagesPullingTime="0s" totalInitContainerRuntime="735.680479ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:16:31.487860263 +0000 UTC m=+2518.244268325" watchObservedRunningTime="2026-07-16 18:16:31.488161967 +0000 UTC m=+2518.244570052"
Jul 16 18:16:43.002074 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:43.002041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:16:51.424355 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:16:51.424324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:17:02.476144 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:17:02.476118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:17:12.822824 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:17:12.822790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:19:51.249766 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:51.249729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:19:51.295037 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:51.294997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:19:51.379496 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:51.379450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:19:52.476372 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:52.476333    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:19:57.168478 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:57.168424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:19:57.189545 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:57.189490    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:19:57.238407 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:19:57.238378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:01.425050 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:01.425016    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:20:02.476328 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:02.476295    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:20:06.271880 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.271814    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main" containerID="cri-o://2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51" gracePeriod=45
Jul 16 18:20:06.530193 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.530134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:20:06.713955 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.713925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location\") pod \"012fda12-abfb-4048-a270-ded28ced8ec7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") "
Jul 16 18:20:06.713955 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.713956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs\") pod \"012fda12-abfb-4048-a270-ded28ced8ec7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") "
Jul 16 18:20:06.714155 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.713991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds\") pod \"012fda12-abfb-4048-a270-ded28ced8ec7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") "
Jul 16 18:20:06.714155 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.714039    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spvzv\" (UniqueName: \"kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv\") pod \"012fda12-abfb-4048-a270-ded28ced8ec7\" (UID: \"012fda12-abfb-4048-a270-ded28ced8ec7\") "
Jul 16 18:20:06.714312 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.714291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "012fda12-abfb-4048-a270-ded28ced8ec7" (UID: "012fda12-abfb-4048-a270-ded28ced8ec7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:06.714645 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.714621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "012fda12-abfb-4048-a270-ded28ced8ec7" (UID: "012fda12-abfb-4048-a270-ded28ced8ec7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:06.716021 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.716000    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv" (OuterVolumeSpecName: "kube-api-access-spvzv") pod "012fda12-abfb-4048-a270-ded28ced8ec7" (UID: "012fda12-abfb-4048-a270-ded28ced8ec7"). InnerVolumeSpecName "kube-api-access-spvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:06.716082 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.716066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "012fda12-abfb-4048-a270-ded28ced8ec7" (UID: "012fda12-abfb-4048-a270-ded28ced8ec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:20:06.815194 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.815135    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:06.815194 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.815155    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fda12-abfb-4048-a270-ded28ced8ec7-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:06.815194 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.815165    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/012fda12-abfb-4048-a270-ded28ced8ec7-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:06.815194 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:06.815173    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvzv\" (UniqueName: \"kubernetes.io/projected/012fda12-abfb-4048-a270-ded28ced8ec7-kube-api-access-spvzv\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.320467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.320433    2577 generic.go:356] "Generic (PLEG): container finished" podID="012fda12-abfb-4048-a270-ded28ced8ec7" containerID="2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51" exitCode=0
Jul 16 18:20:07.320953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.320491    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"
Jul 16 18:20:07.320953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.320496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerDied","Data":"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"}
Jul 16 18:20:07.320953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.320551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7" event={"ID":"012fda12-abfb-4048-a270-ded28ced8ec7","Type":"ContainerDied","Data":"5eabc438c201ce48f321420665a429e861af7096fc50949f2ac80b4170fb7de7"}
Jul 16 18:20:07.320953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.320573    2577 scope.go:117] "RemoveContainer" containerID="2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"
Jul 16 18:20:07.329233 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.329210    2577 scope.go:117] "RemoveContainer" containerID="e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"
Jul 16 18:20:07.336846 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.336827    2577 scope.go:117] "RemoveContainer" containerID="2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"
Jul 16 18:20:07.337103 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:20:07.337079    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51\": container with ID starting with 2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51 not found: ID does not exist" containerID="2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"
Jul 16 18:20:07.337183 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.337114    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51"} err="failed to get container status \"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51\": rpc error: code = NotFound desc = could not find container \"2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51\": container with ID starting with 2afa8bad0f16e407ccd55c81118518968085f8c4d84f46015844660e2ac93e51 not found: ID does not exist"
Jul 16 18:20:07.337183 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.337139    2577 scope.go:117] "RemoveContainer" containerID="e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"
Jul 16 18:20:07.337411 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:20:07.337391    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769\": container with ID starting with e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769 not found: ID does not exist" containerID="e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"
Jul 16 18:20:07.337467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.337416    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769"} err="failed to get container status \"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769\": rpc error: code = NotFound desc = could not find container \"e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769\": container with ID starting with e3144df5fd166d474f781c53f53a7dcf71ae640921d407ff5b9d3f43e9b3f769 not found: ID does not exist"
Jul 16 18:20:07.345601 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.345580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:20:07.350927 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:07.350910    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-546967d655-nrcg7"]
Jul 16 18:20:08.304805 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:08.304773    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" path="/var/lib/kubelet/pods/012fda12-abfb-4048-a270-ded28ced8ec7/volumes"
Jul 16 18:20:11.424239 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:11.424201    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:20:12.209491 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.209453    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main" containerID="cri-o://7fbab6903b1a1e4bec9ca555e58bdf7bbfce73d69c8beb656d574a137eb53a00" gracePeriod=45
Jul 16 18:20:12.342493 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.342459    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerID="7fbab6903b1a1e4bec9ca555e58bdf7bbfce73d69c8beb656d574a137eb53a00" exitCode=0
Jul 16 18:20:12.342612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.342487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" event={"ID":"f9818ffb-91f0-47b0-ae3e-1ded1f490e26","Type":"ContainerDied","Data":"7fbab6903b1a1e4bec9ca555e58bdf7bbfce73d69c8beb656d574a137eb53a00"}
Jul 16 18:20:12.455356 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.455336    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:20:12.559566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.559546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4sqk\" (UniqueName: \"kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk\") pod \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") "
Jul 16 18:20:12.559705 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.559580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs\") pod \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") "
Jul 16 18:20:12.559781 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.559720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location\") pod \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") "
Jul 16 18:20:12.559847 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.559811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds\") pod \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\" (UID: \"f9818ffb-91f0-47b0-ae3e-1ded1f490e26\") "
Jul 16 18:20:12.560132 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.560107    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f9818ffb-91f0-47b0-ae3e-1ded1f490e26" (UID: "f9818ffb-91f0-47b0-ae3e-1ded1f490e26"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:12.560350 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.560329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9818ffb-91f0-47b0-ae3e-1ded1f490e26" (UID: "f9818ffb-91f0-47b0-ae3e-1ded1f490e26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:12.561569 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.561549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk" (OuterVolumeSpecName: "kube-api-access-b4sqk") pod "f9818ffb-91f0-47b0-ae3e-1ded1f490e26" (UID: "f9818ffb-91f0-47b0-ae3e-1ded1f490e26"). InnerVolumeSpecName "kube-api-access-b4sqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:12.561624 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.561563    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9818ffb-91f0-47b0-ae3e-1ded1f490e26" (UID: "f9818ffb-91f0-47b0-ae3e-1ded1f490e26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:20:12.660794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.660764    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:12.660794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.660794    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:12.660924 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.660805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4sqk\" (UniqueName: \"kubernetes.io/projected/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-kube-api-access-b4sqk\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:12.660924 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:12.660814    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9818ffb-91f0-47b0-ae3e-1ded1f490e26-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.348019 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.347991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x" event={"ID":"f9818ffb-91f0-47b0-ae3e-1ded1f490e26","Type":"ContainerDied","Data":"b7206ae68754c3d6b91f3190b48dc36e17de2ef79d6871ad63c12ca7b956f938"}
Jul 16 18:20:13.348196 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.348028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"
Jul 16 18:20:13.348196 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.348043    2577 scope.go:117] "RemoveContainer" containerID="7fbab6903b1a1e4bec9ca555e58bdf7bbfce73d69c8beb656d574a137eb53a00"
Jul 16 18:20:13.356710 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.356675    2577 scope.go:117] "RemoveContainer" containerID="c6d03e0045bfb7aaf07eccf7e6d9f235a4737c7caef64945dd6db58375e2e454"
Jul 16 18:20:13.373092 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.373066    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:20:13.378402 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:13.378383    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-548f4klq5x"]
Jul 16 18:20:14.305409 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:14.305377    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" path="/var/lib/kubelet/pods/f9818ffb-91f0-47b0-ae3e-1ded1f490e26/volumes"
Jul 16 18:20:14.647865 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:14.647770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:18.184004 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.183971    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:18.184405 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184391    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main"
Jul 16 18:20:18.184445 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184409    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main"
Jul 16 18:20:18.184445 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="storage-initializer"
Jul 16 18:20:18.184534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184450    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="storage-initializer"
Jul 16 18:20:18.184534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184462    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="storage-initializer"
Jul 16 18:20:18.184534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184471    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="storage-initializer"
Jul 16 18:20:18.184534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main"
Jul 16 18:20:18.184534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184489    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main"
Jul 16 18:20:18.184687 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184568    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9818ffb-91f0-47b0-ae3e-1ded1f490e26" containerName="main"
Jul 16 18:20:18.184687 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.184580    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="012fda12-abfb-4048-a270-ded28ced8ec7" containerName="main"
Jul 16 18:20:18.189162 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.189124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.196095 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.196070    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:20:18.196706 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.196687    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:20:18.196806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.196689    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 16 18:20:18.196806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.196729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-f4wl5\""
Jul 16 18:20:18.201060 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfxm\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201145 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201145 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201145 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201256 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201256 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201256 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201256 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.201394 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.201288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.207292 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.207263    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:18.220404 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.220382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:18.287555 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.287523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:18.302857 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.302829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.302974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.302906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.302974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.302939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303082 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.302997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303082 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfxm\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303082 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303235 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303235 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303235 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.303647 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.303625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.304805 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.304773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.305048 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.305025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.305287 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.305268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.307393 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.305722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.307393 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.306881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.308970 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.307800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.314515 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.314488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfxm\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.314685 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.314669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-44z29\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.499306 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.499273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:18.637606 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.637574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:18.638971 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:20:18.638946    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c29657_10b0_400e_8535_53419ce9563e.slice/crio-3e89b4c758b101bdf6e8ed47bb7efe28269f2c1b440cc666e8f645b9970b6082 WatchSource:0}: Error finding container 3e89b4c758b101bdf6e8ed47bb7efe28269f2c1b440cc666e8f645b9970b6082: Status 404 returned error can't find the container with id 3e89b4c758b101bdf6e8ed47bb7efe28269f2c1b440cc666e8f645b9970b6082
Jul 16 18:20:18.640917 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.640879    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:18.641006 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.640960    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:18.641006 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:18.640998    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:19.372681 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:19.372642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" event={"ID":"85c29657-10b0-400e-8535-53419ce9563e","Type":"ContainerStarted","Data":"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"}
Jul 16 18:20:19.373014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:19.372697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" event={"ID":"85c29657-10b0-400e-8535-53419ce9563e","Type":"ContainerStarted","Data":"3e89b4c758b101bdf6e8ed47bb7efe28269f2c1b440cc666e8f645b9970b6082"}
Jul 16 18:20:19.398863 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:19.398817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" podStartSLOduration=1.39880287 podStartE2EDuration="1.39880287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:19.398208895 +0000 UTC m=+2746.154616957" watchObservedRunningTime="2026-07-16 18:20:19.39880287 +0000 UTC m=+2746.155210931"
Jul 16 18:20:19.499704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:19.499669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:19.504408 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:19.504384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:20.376482 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.376449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:20.377411 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.377393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:20.909956 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.909924    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"]
Jul 16 18:20:20.913207 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.913190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924323 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924434 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924434 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924434 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924637 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924668 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt8p\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-kube-api-access-6lt8p\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924701 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:20.924830 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:20.924816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"]
Jul 16 18:20:21.025630 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.025765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.025765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.025765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.025765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.025765 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.025942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026127 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt8p\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-kube-api-access-6lt8p\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026127 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026127 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026258 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026434 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.026493 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.026470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.028152 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.028130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.028257 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.028237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.036089 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.036068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.036178 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.036117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt8p\" (UniqueName: \"kubernetes.io/projected/811b18c8-7bd3-4780-8c4f-7662fbfd21e8-kube-api-access-6lt8p\") pod \"router-gateway-2-openshift-default-684dc9b4b9-zbq44\" (UID: \"811b18c8-7bd3-4780-8c4f-7662fbfd21e8\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.222845 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.222763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:21.361747 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.361614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"]
Jul 16 18:20:21.364171 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:20:21.364144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811b18c8_7bd3_4780_8c4f_7662fbfd21e8.slice/crio-0dcb101725453d0b2f4376b63c0e99891d7a8528b4c3018cf14100cc4717e73f WatchSource:0}: Error finding container 0dcb101725453d0b2f4376b63c0e99891d7a8528b4c3018cf14100cc4717e73f: Status 404 returned error can't find the container with id 0dcb101725453d0b2f4376b63c0e99891d7a8528b4c3018cf14100cc4717e73f
Jul 16 18:20:21.366265 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.366222    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:21.366345 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.366306    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:21.366387 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.366348    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:21.381724 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:21.381701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" event={"ID":"811b18c8-7bd3-4780-8c4f-7662fbfd21e8","Type":"ContainerStarted","Data":"0dcb101725453d0b2f4376b63c0e99891d7a8528b4c3018cf14100cc4717e73f"}
Jul 16 18:20:22.018190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:22.018154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:22.387442 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:22.387355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" event={"ID":"811b18c8-7bd3-4780-8c4f-7662fbfd21e8","Type":"ContainerStarted","Data":"f0d712788680edd4515af0647752ea731705b8b166a48fea70723ee53d3b291a"}
Jul 16 18:20:22.414205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:22.414154    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podStartSLOduration=2.414139359 podStartE2EDuration="2.414139359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:22.41194226 +0000 UTC m=+2749.168350323" watchObservedRunningTime="2026-07-16 18:20:22.414139359 +0000 UTC m=+2749.170547421"
Jul 16 18:20:23.223264 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.223229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:23.227487 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.227463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:23.391908 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.391877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:23.392792 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.392772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:23.446742 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.446713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:23.446912 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:23.446894    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" podUID="85c29657-10b0-400e-8535-53419ce9563e" containerName="istio-proxy" containerID="cri-o://a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384" gracePeriod=30
Jul 16 18:20:28.703785 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.703762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:28.793562 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hfxm\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.793903 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:28.794110 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.794110 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:28.794110 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.793994    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket\") pod \"85c29657-10b0-400e-8535-53419ce9563e\" (UID: \"85c29657-10b0-400e-8535-53419ce9563e\") "
Jul 16 18:20:28.794110 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data" (OuterVolumeSpecName: "istio-data") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:28.794303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:20:28.794303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:28.794303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794288    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-credential-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.794472 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794307    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.794472 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794338    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-data\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.794472 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.794353    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c29657-10b0-400e-8535-53419ce9563e-istiod-ca-cert\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.795743 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.795710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm" (OuterVolumeSpecName: "kube-api-access-8hfxm") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "kube-api-access-8hfxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:28.796030 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.796005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token" (OuterVolumeSpecName: "istio-token") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:28.796198 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.796179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 18:20:28.796242 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.796180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85c29657-10b0-400e-8535-53419ce9563e" (UID: "85c29657-10b0-400e-8535-53419ce9563e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:28.894798 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.894773    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c29657-10b0-400e-8535-53419ce9563e-istio-podinfo\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.894798 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.894798    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-istio-envoy\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.894972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.894810    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c29657-10b0-400e-8535-53419ce9563e-workload-socket\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.894972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.894821    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-istio-token\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:28.894972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:28.894832    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfxm\" (UniqueName: \"kubernetes.io/projected/85c29657-10b0-400e-8535-53419ce9563e-kube-api-access-8hfxm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:20:29.418603 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.418564    2577 generic.go:356] "Generic (PLEG): container finished" podID="85c29657-10b0-400e-8535-53419ce9563e" containerID="a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384" exitCode=0
Jul 16 18:20:29.418603 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.418607    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"
Jul 16 18:20:29.418806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.418640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" event={"ID":"85c29657-10b0-400e-8535-53419ce9563e","Type":"ContainerDied","Data":"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"}
Jul 16 18:20:29.418806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.418684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29" event={"ID":"85c29657-10b0-400e-8535-53419ce9563e","Type":"ContainerDied","Data":"3e89b4c758b101bdf6e8ed47bb7efe28269f2c1b440cc666e8f645b9970b6082"}
Jul 16 18:20:29.418806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.418708    2577 scope.go:117] "RemoveContainer" containerID="a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"
Jul 16 18:20:29.437031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.436936    2577 scope.go:117] "RemoveContainer" containerID="a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"
Jul 16 18:20:29.437338 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:20:29.437265    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384\": container with ID starting with a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384 not found: ID does not exist" containerID="a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"
Jul 16 18:20:29.437338 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.437303    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384"} err="failed to get container status \"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384\": rpc error: code = NotFound desc = could not find container \"a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384\": container with ID starting with a291f74169c4c71cd259b8fc9c7e11773679f5c22142f2d0000cee7cc594a384 not found: ID does not exist"
Jul 16 18:20:29.443654 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.443630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:29.459327 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:29.459308    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-44z29"]
Jul 16 18:20:30.303616 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:30.303590    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c29657-10b0-400e-8535-53419ce9563e" path="/var/lib/kubelet/pods/85c29657-10b0-400e-8535-53419ce9563e/volumes"
Jul 16 18:20:33.438532 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.438475    2577 generic.go:356] "Generic (PLEG): container finished" podID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerID="f0d712788680edd4515af0647752ea731705b8b166a48fea70723ee53d3b291a" exitCode=0
Jul 16 18:20:33.439018 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.438544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" event={"ID":"811b18c8-7bd3-4780-8c4f-7662fbfd21e8","Type":"ContainerDied","Data":"f0d712788680edd4515af0647752ea731705b8b166a48fea70723ee53d3b291a"}
Jul 16 18:20:33.439090 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.439026    2577 scope.go:117] "RemoveContainer" containerID="f0d712788680edd4515af0647752ea731705b8b166a48fea70723ee53d3b291a"
Jul 16 18:20:33.439927 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.439904    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:33.439983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.439970    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:33.440018 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:33.440002    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 16 18:20:34.445120 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:34.445093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" event={"ID":"811b18c8-7bd3-4780-8c4f-7662fbfd21e8","Type":"ContainerStarted","Data":"c059cd6d889f9114038bd070733dd28b091c9ecb7ca2ed63b1a04ec0ddcf0cd3"}
Jul 16 18:20:35.223712 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:35.223681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:35.224808 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:35.224768    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podUID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 16 18:20:36.223467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:36.223428    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podUID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 16 18:20:37.223540 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:37.223477    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podUID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 16 18:20:38.223347 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:38.223308    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podUID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 16 18:20:38.392586 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:38.392549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:39.223111 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:39.223072    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44" podUID="811b18c8-7bd3-4780-8c4f-7662fbfd21e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 16 18:20:40.227367 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:40.227339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:40.228308 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:40.228288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-zbq44"
Jul 16 18:20:40.932751 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:40.932711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:41.699366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.699334    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:20:41.699731 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.699641    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c29657-10b0-400e-8535-53419ce9563e" containerName="istio-proxy"
Jul 16 18:20:41.699731 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.699655    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c29657-10b0-400e-8535-53419ce9563e" containerName="istio-proxy"
Jul 16 18:20:41.699731 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.699725    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c29657-10b0-400e-8535-53419ce9563e" containerName="istio-proxy"
Jul 16 18:20:41.702682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.702666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.708209 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.708192    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:20:41.709419 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.709403    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 16 18:20:41.710311 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.710296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:20:41.718511 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.718487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-67lwj\""
Jul 16 18:20:41.753302 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.753280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:41.804837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.804811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.804943 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.804849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94dd\" (UniqueName: \"kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.804943 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.804870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.805031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.805008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.859841 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.859801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:20:41.894554 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.894514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:20:41.906479 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.906638 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94dd\" (UniqueName: \"kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.906702 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.906802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.906914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.907036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.906955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.909564 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.909544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:41.919900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:41.919874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94dd\" (UniqueName: \"kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:42.011953 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:42.011931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:42.139016 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:42.138928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:20:42.141146 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:20:42.141121    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode270bf06_d090_48ff_acbb_0fe8d4719134.slice/crio-9a02bcd37968abade9ffb1a49c0b6fe4465568a0e6aeb0ffe72b0a57eb6bd130 WatchSource:0}: Error finding container 9a02bcd37968abade9ffb1a49c0b6fe4465568a0e6aeb0ffe72b0a57eb6bd130: Status 404 returned error can't find the container with id 9a02bcd37968abade9ffb1a49c0b6fe4465568a0e6aeb0ffe72b0a57eb6bd130
Jul 16 18:20:42.478689 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:42.478596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerStarted","Data":"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"}
Jul 16 18:20:42.478689 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:42.478652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerStarted","Data":"9a02bcd37968abade9ffb1a49c0b6fe4465568a0e6aeb0ffe72b0a57eb6bd130"}
Jul 16 18:20:43.484174 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:43.484131    2577 generic.go:356] "Generic (PLEG): container finished" podID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerID="ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799" exitCode=0
Jul 16 18:20:43.484612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:43.484199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerDied","Data":"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"}
Jul 16 18:20:44.489482 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:44.489449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerStarted","Data":"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"}
Jul 16 18:20:44.489843 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:44.489581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:20:44.512822 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:20:44.512774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" podStartSLOduration=2.780104611 podStartE2EDuration="3.512748544s" totalImagesPullingTime="0s" totalInitContainerRuntime="732.643933ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:44.512214801 +0000 UTC m=+2771.268622865" watchObservedRunningTime="2026-07-16 18:20:44.512748544 +0000 UTC m=+2771.269156606"
Jul 16 18:21:07.667254 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:21:07.667168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:21:15.495194 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:21:15.495165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:22:25.288414 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:22:25.288330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:22:42.497890 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:22:42.497858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:23:14.109100 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:23:14.109064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:24:01.825184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:01.825090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:24:01.825730 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:01.825477    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:24:01.885709 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:01.885677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:24:05.495534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:05.495481    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:24:15.495359 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:15.495316    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:24:16.845030 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:16.844989    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main" containerID="cri-o://4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4" gracePeriod=45
Jul 16 18:24:17.088251 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.088222    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:24:17.119155 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds\") pod \"e270bf06-d090-48ff-acbb-0fe8d4719134\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") "
Jul 16 18:24:17.119155 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q94dd\" (UniqueName: \"kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd\") pod \"e270bf06-d090-48ff-acbb-0fe8d4719134\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") "
Jul 16 18:24:17.119335 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs\") pod \"e270bf06-d090-48ff-acbb-0fe8d4719134\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") "
Jul 16 18:24:17.119335 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location\") pod \"e270bf06-d090-48ff-acbb-0fe8d4719134\" (UID: \"e270bf06-d090-48ff-acbb-0fe8d4719134\") "
Jul 16 18:24:17.119441 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e270bf06-d090-48ff-acbb-0fe8d4719134" (UID: "e270bf06-d090-48ff-acbb-0fe8d4719134"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:17.119603 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.119587    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:24:17.120033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.120011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e270bf06-d090-48ff-acbb-0fe8d4719134" (UID: "e270bf06-d090-48ff-acbb-0fe8d4719134"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:17.121307 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.121281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e270bf06-d090-48ff-acbb-0fe8d4719134" (UID: "e270bf06-d090-48ff-acbb-0fe8d4719134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:24:17.121413 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.121325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd" (OuterVolumeSpecName: "kube-api-access-q94dd") pod "e270bf06-d090-48ff-acbb-0fe8d4719134" (UID: "e270bf06-d090-48ff-acbb-0fe8d4719134"). InnerVolumeSpecName "kube-api-access-q94dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:24:17.220259 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.220230    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94dd\" (UniqueName: \"kubernetes.io/projected/e270bf06-d090-48ff-acbb-0fe8d4719134-kube-api-access-q94dd\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:24:17.220259 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.220254    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270bf06-d090-48ff-acbb-0fe8d4719134-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:24:17.220259 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.220266    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e270bf06-d090-48ff-acbb-0fe8d4719134-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:24:17.322688 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.322655    2577 generic.go:356] "Generic (PLEG): container finished" podID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerID="4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4" exitCode=0
Jul 16 18:24:17.322837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.322701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"
Jul 16 18:24:17.322837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.322708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerDied","Data":"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"}
Jul 16 18:24:17.322837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.322756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f" event={"ID":"e270bf06-d090-48ff-acbb-0fe8d4719134","Type":"ContainerDied","Data":"9a02bcd37968abade9ffb1a49c0b6fe4465568a0e6aeb0ffe72b0a57eb6bd130"}
Jul 16 18:24:17.322837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.322785    2577 scope.go:117] "RemoveContainer" containerID="4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"
Jul 16 18:24:17.331368 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.331346    2577 scope.go:117] "RemoveContainer" containerID="ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"
Jul 16 18:24:17.338468 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.338453    2577 scope.go:117] "RemoveContainer" containerID="4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"
Jul 16 18:24:17.338748 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:24:17.338730    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4\": container with ID starting with 4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4 not found: ID does not exist" containerID="4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"
Jul 16 18:24:17.338809 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.338755    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4"} err="failed to get container status \"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4\": rpc error: code = NotFound desc = could not find container \"4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4\": container with ID starting with 4e416e6a0e4085a8173f2883d5ec094a4ccd644b2279a529f5a412a4ffc89df4 not found: ID does not exist"
Jul 16 18:24:17.338809 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.338772    2577 scope.go:117] "RemoveContainer" containerID="ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"
Jul 16 18:24:17.339039 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:24:17.339017    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799\": container with ID starting with ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799 not found: ID does not exist" containerID="ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"
Jul 16 18:24:17.339093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.339052    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799"} err="failed to get container status \"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799\": rpc error: code = NotFound desc = could not find container \"ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799\": container with ID starting with ecc3b47d389ea8d7518d83a04c9bf9e55b3ed5bbf9304a0fdbccf00ce79c2799 not found: ID does not exist"
Jul 16 18:24:17.362305 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.362280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:24:17.377518 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:17.377448    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6668cbdbdh7z6f"]
Jul 16 18:24:18.305195 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:18.305161    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" path="/var/lib/kubelet/pods/e270bf06-d090-48ff-acbb-0fe8d4719134/volumes"
Jul 16 18:24:26.003389 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:24:26.003351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:25:43.201804 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:25:43.201766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:26:21.808289 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:21.808254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:26:22.175102 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175014    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:26:22.175370 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175357    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main"
Jul 16 18:26:22.175419 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175372    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main"
Jul 16 18:26:22.175419 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175380    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="storage-initializer"
Jul 16 18:26:22.175419 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="storage-initializer"
Jul 16 18:26:22.175556 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.175450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e270bf06-d090-48ff-acbb-0fe8d4719134" containerName="main"
Jul 16 18:26:22.178368 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.178352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.181019 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.180995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:26:22.181147 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.181047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:26:22.181147 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.181092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-72zdr\""
Jul 16 18:26:22.181913 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.181891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 16 18:26:22.189080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.189060    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:26:22.208579 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.208544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:26:22.292857 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.292823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.293031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.292887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.293031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.292906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.293031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.292936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hmv\" (UniqueName: \"kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.393645 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.393605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hmv\" (UniqueName: \"kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.393829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.393660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.393829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.393711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.393829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.393730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.394094 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.394073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.394134 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.394080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.396224 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.396200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.403145 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.403114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hmv\" (UniqueName: \"kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv\") pod \"tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.488945 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.488854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:22.622033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.621997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:26:22.625792 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:26:22.625763    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb88bbc9_2f5a_4321_8020_307a183e98b5.slice/crio-dff15c1e2999e68c77f5e239dfae3143e9fd9e7351cf0f7beb23118b59923cbd WatchSource:0}: Error finding container dff15c1e2999e68c77f5e239dfae3143e9fd9e7351cf0f7beb23118b59923cbd: Status 404 returned error can't find the container with id dff15c1e2999e68c77f5e239dfae3143e9fd9e7351cf0f7beb23118b59923cbd
Jul 16 18:26:22.800763 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.800728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerStarted","Data":"1dc9cab6c8d814035d4fe8423f2bcfdc2a21a1a0f6c35f9d588b7c6cf82c8c7e"}
Jul 16 18:26:22.800931 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:22.800780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerStarted","Data":"dff15c1e2999e68c77f5e239dfae3143e9fd9e7351cf0f7beb23118b59923cbd"}
Jul 16 18:26:23.805623 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:23.805589    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerID="1dc9cab6c8d814035d4fe8423f2bcfdc2a21a1a0f6c35f9d588b7c6cf82c8c7e" exitCode=0
Jul 16 18:26:23.805961 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:23.805641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerDied","Data":"1dc9cab6c8d814035d4fe8423f2bcfdc2a21a1a0f6c35f9d588b7c6cf82c8c7e"}
Jul 16 18:26:24.811513 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:24.811463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerStarted","Data":"6c7b94f9e28c8b048543b96b20e5c3d70a8803f0d586799647c3edf591d4797f"}
Jul 16 18:26:24.811881 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:24.811542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:26:24.855261 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:24.855213    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" podStartSLOduration=2.080236827 podStartE2EDuration="2.855195289s" totalImagesPullingTime="0s" totalInitContainerRuntime="774.958462ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:26:24.853550182 +0000 UTC m=+3111.609958245" watchObservedRunningTime="2026-07-16 18:26:24.855195289 +0000 UTC m=+3111.611603352"
Jul 16 18:26:28.714710 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:28.714677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:26:50.826354 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:50.826317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:26:55.817863 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:26:55.817779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:27:28.128139 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.128100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:27:28.554296 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.554258    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:27:28.557748 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.557728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.560190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.560161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 16 18:27:28.560290 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.560220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-tc8lc\""
Jul 16 18:27:28.570000 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.569975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:27:28.686494 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.686460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.686682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.686498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsmh\" (UniqueName: \"kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.686682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.686575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.686682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.686642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.787644 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.787617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.787785 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.787656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsmh\" (UniqueName: \"kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.787785 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.787701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.787785 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.787733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.788166 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.788146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.788236 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.788164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.790150 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.790128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.796895 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.796867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsmh\" (UniqueName: \"kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.866387 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.866332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:28.995341 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:28.995313    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:27:28.998180 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:27:28.998154    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc35cb46_e2dc_4c59_a6f1_a7a7b0dede13.slice/crio-4986786afeb9e4dfee6a2078f16cd409cbd8be24c762b62b5d99525c3cfe24f4 WatchSource:0}: Error finding container 4986786afeb9e4dfee6a2078f16cd409cbd8be24c762b62b5d99525c3cfe24f4: Status 404 returned error can't find the container with id 4986786afeb9e4dfee6a2078f16cd409cbd8be24c762b62b5d99525c3cfe24f4
Jul 16 18:27:29.058715 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:29.058689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerStarted","Data":"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"}
Jul 16 18:27:29.058833 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:29.058734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerStarted","Data":"4986786afeb9e4dfee6a2078f16cd409cbd8be24c762b62b5d99525c3cfe24f4"}
Jul 16 18:27:30.064585 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:30.064479    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerID="f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8" exitCode=0
Jul 16 18:27:30.064585 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:30.064566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerDied","Data":"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"}
Jul 16 18:27:31.070660 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:31.070627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerStarted","Data":"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"}
Jul 16 18:27:31.071012 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:31.070699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:27:31.092171 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:31.092127    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" podStartSLOduration=2.3376572700000002 podStartE2EDuration="3.092113247s" totalImagesPullingTime="0s" totalInitContainerRuntime="754.455977ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:27:31.091087379 +0000 UTC m=+3177.847495442" watchObservedRunningTime="2026-07-16 18:27:31.092113247 +0000 UTC m=+3177.848521309"
Jul 16 18:27:31.318901 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:31.318873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:27:38.266053 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:38.266023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:27:53.111206 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:53.111170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:27:55.719574 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:27:55.719530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:02.076906 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:02.076874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:28:11.839156 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:11.839116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:27.198793 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:27.198705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:27.199542 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:27.199520    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:28:27.292660 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:27.292618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:35.817332 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:35.817292    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:28:42.218870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.218809    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="main" containerID="cri-o://6c7b94f9e28c8b048543b96b20e5c3d70a8803f0d586799647c3edf591d4797f" gracePeriod=45
Jul 16 18:28:42.382331 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.382302    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerID="6c7b94f9e28c8b048543b96b20e5c3d70a8803f0d586799647c3edf591d4797f" exitCode=0
Jul 16 18:28:42.382468 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.382354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerDied","Data":"6c7b94f9e28c8b048543b96b20e5c3d70a8803f0d586799647c3edf591d4797f"}
Jul 16 18:28:42.467766 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.467745    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:28:42.565855 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.565828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5hmv\" (UniqueName: \"kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv\") pod \"eb88bbc9-2f5a-4321-8020-307a183e98b5\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") "
Jul 16 18:28:42.565983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.565882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds\") pod \"eb88bbc9-2f5a-4321-8020-307a183e98b5\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") "
Jul 16 18:28:42.565983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.565917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs\") pod \"eb88bbc9-2f5a-4321-8020-307a183e98b5\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") "
Jul 16 18:28:42.565983 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.565952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location\") pod \"eb88bbc9-2f5a-4321-8020-307a183e98b5\" (UID: \"eb88bbc9-2f5a-4321-8020-307a183e98b5\") "
Jul 16 18:28:42.566223 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.566198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "eb88bbc9-2f5a-4321-8020-307a183e98b5" (UID: "eb88bbc9-2f5a-4321-8020-307a183e98b5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:42.566639 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.566620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb88bbc9-2f5a-4321-8020-307a183e98b5" (UID: "eb88bbc9-2f5a-4321-8020-307a183e98b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:42.567906 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.567884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv" (OuterVolumeSpecName: "kube-api-access-t5hmv") pod "eb88bbc9-2f5a-4321-8020-307a183e98b5" (UID: "eb88bbc9-2f5a-4321-8020-307a183e98b5"). InnerVolumeSpecName "kube-api-access-t5hmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:28:42.567995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.567907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb88bbc9-2f5a-4321-8020-307a183e98b5" (UID: "eb88bbc9-2f5a-4321-8020-307a183e98b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:28:42.666677 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.666655    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hmv\" (UniqueName: \"kubernetes.io/projected/eb88bbc9-2f5a-4321-8020-307a183e98b5-kube-api-access-t5hmv\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:28:42.666677 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.666676    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:28:42.666806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.666686    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb88bbc9-2f5a-4321-8020-307a183e98b5-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:28:42.666806 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:42.666695    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb88bbc9-2f5a-4321-8020-307a183e98b5-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.388711 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.388686    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"
Jul 16 18:28:43.389085 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.388716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg" event={"ID":"eb88bbc9-2f5a-4321-8020-307a183e98b5","Type":"ContainerDied","Data":"dff15c1e2999e68c77f5e239dfae3143e9fd9e7351cf0f7beb23118b59923cbd"}
Jul 16 18:28:43.389085 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.388769    2577 scope.go:117] "RemoveContainer" containerID="6c7b94f9e28c8b048543b96b20e5c3d70a8803f0d586799647c3edf591d4797f"
Jul 16 18:28:43.398087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.398069    2577 scope.go:117] "RemoveContainer" containerID="1dc9cab6c8d814035d4fe8423f2bcfdc2a21a1a0f6c35f9d588b7c6cf82c8c7e"
Jul 16 18:28:43.412035 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.412010    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:28:43.416432 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:43.416410    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-59969bd6f5-4v7gg"]
Jul 16 18:28:44.311696 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:44.311662    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" path="/var/lib/kubelet/pods/eb88bbc9-2f5a-4321-8020-307a183e98b5/volumes"
Jul 16 18:28:50.399891 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.399851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:50.817118 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:28:50.817378 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817365    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="storage-initializer"
Jul 16 18:28:50.817378 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817378    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="storage-initializer"
Jul 16 18:28:50.817465 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="main"
Jul 16 18:28:50.817465 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817409    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="main"
Jul 16 18:28:50.817465 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.817461    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb88bbc9-2f5a-4321-8020-307a183e98b5" containerName="main"
Jul 16 18:28:50.820353 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.820332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:50.822816 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.822795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-8fq5p\""
Jul 16 18:28:50.822896 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.822795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 16 18:28:50.831798 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.831777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:28:50.883353 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.883323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:28:50.924397 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.924370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:50.924550 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.924412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fx4\" (UniqueName: \"kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:50.924550 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.924478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:50.924716 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:50.924553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.025692 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.025661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.025810 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.025754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.025810 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.025791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fx4\" (UniqueName: \"kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.025936 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.025836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.026101 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.026077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.026172 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.026151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.028198 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.028175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.034692 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.034674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fx4\" (UniqueName: \"kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4\") pod \"prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.129801 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.129742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:51.261248 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.261218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:28:51.263083 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:28:51.263056    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e48cc0_8608_47fb_b9f8_f08ba419b83f.slice/crio-d5c42d8bb14f8bbd9153fa7e2afa58615e2cc97183799aede3152bad74564194 WatchSource:0}: Error finding container d5c42d8bb14f8bbd9153fa7e2afa58615e2cc97183799aede3152bad74564194: Status 404 returned error can't find the container with id d5c42d8bb14f8bbd9153fa7e2afa58615e2cc97183799aede3152bad74564194
Jul 16 18:28:51.427564 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.427458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerStarted","Data":"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"}
Jul 16 18:28:51.427564 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:51.427541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerStarted","Data":"d5c42d8bb14f8bbd9153fa7e2afa58615e2cc97183799aede3152bad74564194"}
Jul 16 18:28:52.433473 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:52.433395    2577 generic.go:356] "Generic (PLEG): container finished" podID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerID="3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a" exitCode=0
Jul 16 18:28:52.433817 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:52.433469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerDied","Data":"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"}
Jul 16 18:28:53.439768 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:53.439735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerStarted","Data":"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"}
Jul 16 18:28:53.440144 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:53.439840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:28:53.460796 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:28:53.460756    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" podStartSLOduration=2.7107058889999998 podStartE2EDuration="3.46073874s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.032851ms" isStatefulPod=false podCreationTimestamp="2026-07-16 18:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:28:53.459042243 +0000 UTC m=+3260.215450305" watchObservedRunningTime="2026-07-16 18:28:53.46073874 +0000 UTC m=+3260.217146804"
Jul 16 18:29:12.355614 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:29:12.355558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:29:24.447661 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:29:24.447636    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:30:16.721937 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:30:16.721829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:30:16.799965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:30:16.799935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:11.790030 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:11.789998    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:31:11.801696 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:11.801669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:11.902029 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:11.901999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:14.447435 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:14.447398    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:31:18.073402 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:18.073363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:18.092524 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:18.092485    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:31:18.135375 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:18.135346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:22.077118 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:22.077080    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:31:24.448300 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:24.448259    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:31:26.810850 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:26.810729    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main" containerID="cri-o://0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603" gracePeriod=45
Jul 16 18:31:27.063712 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.063660    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:31:27.070967 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.070940    2577 generic.go:356] "Generic (PLEG): container finished" podID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerID="0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603" exitCode=0
Jul 16 18:31:27.071054 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.070978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"
Jul 16 18:31:27.071054 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.071029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerDied","Data":"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"}
Jul 16 18:31:27.071129 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.071077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2" event={"ID":"60e48cc0-8608-47fb-b9f8-f08ba419b83f","Type":"ContainerDied","Data":"d5c42d8bb14f8bbd9153fa7e2afa58615e2cc97183799aede3152bad74564194"}
Jul 16 18:31:27.071129 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.071100    2577 scope.go:117] "RemoveContainer" containerID="0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"
Jul 16 18:31:27.080898 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.080881    2577 scope.go:117] "RemoveContainer" containerID="3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"
Jul 16 18:31:27.089182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.089164    2577 scope.go:117] "RemoveContainer" containerID="0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"
Jul 16 18:31:27.090186 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:31:27.090160    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603\": container with ID starting with 0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603 not found: ID does not exist" containerID="0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"
Jul 16 18:31:27.090292 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.090195    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603"} err="failed to get container status \"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603\": rpc error: code = NotFound desc = could not find container \"0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603\": container with ID starting with 0b79888e6f760c6abeae0a2bbd988f0408c1de1f65db6a4c7ae7573632b4d603 not found: ID does not exist"
Jul 16 18:31:27.090292 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.090225    2577 scope.go:117] "RemoveContainer" containerID="3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"
Jul 16 18:31:27.091556 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:31:27.090725    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a\": container with ID starting with 3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a not found: ID does not exist" containerID="3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"
Jul 16 18:31:27.091556 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.090771    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a"} err="failed to get container status \"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a\": rpc error: code = NotFound desc = could not find container \"3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a\": container with ID starting with 3a12472b9ebd718eb37a36079849ab888f410d53960a4441e284fc0c864b945a not found: ID does not exist"
Jul 16 18:31:27.142943 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.142923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5fx4\" (UniqueName: \"kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4\") pod \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") "
Jul 16 18:31:27.143036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.142964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location\") pod \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") "
Jul 16 18:31:27.143036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.143004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs\") pod \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") "
Jul 16 18:31:27.143036 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.143033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds\") pod \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\" (UID: \"60e48cc0-8608-47fb-b9f8-f08ba419b83f\") "
Jul 16 18:31:27.143341 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.143314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "60e48cc0-8608-47fb-b9f8-f08ba419b83f" (UID: "60e48cc0-8608-47fb-b9f8-f08ba419b83f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:27.143702 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.143681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60e48cc0-8608-47fb-b9f8-f08ba419b83f" (UID: "60e48cc0-8608-47fb-b9f8-f08ba419b83f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:27.144753 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.144733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4" (OuterVolumeSpecName: "kube-api-access-j5fx4") pod "60e48cc0-8608-47fb-b9f8-f08ba419b83f" (UID: "60e48cc0-8608-47fb-b9f8-f08ba419b83f"). InnerVolumeSpecName "kube-api-access-j5fx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:31:27.144857 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.144833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60e48cc0-8608-47fb-b9f8-f08ba419b83f" (UID: "60e48cc0-8608-47fb-b9f8-f08ba419b83f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:31:27.244089 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.244052    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fx4\" (UniqueName: \"kubernetes.io/projected/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kube-api-access-j5fx4\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:27.244089 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.244085    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:27.244089 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.244097    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:27.244279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.244107    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/60e48cc0-8608-47fb-b9f8-f08ba419b83f-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:27.399757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.399729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:31:27.405546 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:27.405522    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6c8475994c-mhqk2"]
Jul 16 18:31:28.304876 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:28.304834    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" path="/var/lib/kubelet/pods/60e48cc0-8608-47fb-b9f8-f08ba419b83f/volumes"
Jul 16 18:31:32.077013 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:32.076978    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:31:33.112444 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.112405    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main" containerID="cri-o://73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e" gracePeriod=45
Jul 16 18:31:33.361838 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.361812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:31:33.485643 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.485566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds\") pod \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") "
Jul 16 18:31:33.485643 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.485605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location\") pod \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") "
Jul 16 18:31:33.485643 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.485633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs\") pod \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") "
Jul 16 18:31:33.485870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.485685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmsmh\" (UniqueName: \"kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh\") pod \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\" (UID: \"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13\") "
Jul 16 18:31:33.485915 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.485853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" (UID: "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:33.486352 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.486327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" (UID: "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:33.487729 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.487709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" (UID: "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:31:33.487833 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.487766    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh" (OuterVolumeSpecName: "kube-api-access-zmsmh") pod "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" (UID: "cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13"). InnerVolumeSpecName "kube-api-access-zmsmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:31:33.587014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.586980    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:33.587014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.587009    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsmh\" (UniqueName: \"kubernetes.io/projected/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kube-api-access-zmsmh\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:33.587014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.587018    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:33.587223 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:33.587027    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13-kserve-provision-location\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.103134 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.103094    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerID="73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e" exitCode=0
Jul 16 18:31:34.103333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.103147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerDied","Data":"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"}
Jul 16 18:31:34.103333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.103226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9" event={"ID":"cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13","Type":"ContainerDied","Data":"4986786afeb9e4dfee6a2078f16cd409cbd8be24c762b62b5d99525c3cfe24f4"}
Jul 16 18:31:34.103333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.103150    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"
Jul 16 18:31:34.103333 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.103244    2577 scope.go:117] "RemoveContainer" containerID="73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"
Jul 16 18:31:34.111716 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.111699    2577 scope.go:117] "RemoveContainer" containerID="f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"
Jul 16 18:31:34.118954 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.118763    2577 scope.go:117] "RemoveContainer" containerID="73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"
Jul 16 18:31:34.119188 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:31:34.118991    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e\": container with ID starting with 73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e not found: ID does not exist" containerID="73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"
Jul 16 18:31:34.119188 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.119010    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e"} err="failed to get container status \"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e\": rpc error: code = NotFound desc = could not find container \"73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e\": container with ID starting with 73b321b469edd15cac06a92d473e16bb38abe6ea6f5014e8f583f5d2096c974e not found: ID does not exist"
Jul 16 18:31:34.119188 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.119034    2577 scope.go:117] "RemoveContainer" containerID="f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"
Jul 16 18:31:34.119335 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:31:34.119272    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8\": container with ID starting with f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8 not found: ID does not exist" containerID="f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"
Jul 16 18:31:34.119335 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.119300    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8"} err="failed to get container status \"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8\": rpc error: code = NotFound desc = could not find container \"f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8\": container with ID starting with f4bcf890818d915023434625a909072aa8d12b4252ddf6b5a253a7524165daa8 not found: ID does not exist"
Jul 16 18:31:34.126474 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.126451    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:31:34.131625 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.131601    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetglm9"]
Jul 16 18:31:34.304764 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:34.304734    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" path="/var/lib/kubelet/pods/cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13/volumes"
Jul 16 18:31:59.895584 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="storage-initializer"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895881    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="storage-initializer"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895902    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895910    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895937    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="storage-initializer"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.895945    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="storage-initializer"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.896013    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e48cc0-8608-47fb-b9f8-f08ba419b83f" containerName="main"
Jul 16 18:31:59.896033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.896026    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc35cb46-e2dc-4c59-a6f1-a7a7b0dede13" containerName="main"
Jul 16 18:31:59.898784 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.898768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.902636 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.902615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:31:59.902766 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.902616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 16 18:31:59.908902 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.908882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:31:59.927344 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.927317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:31:59.983164 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.983253 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rlb\" (UniqueName: \"kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.983253 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.983253 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.983388 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:31:59.983388 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:31:59.983347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084047 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084154 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rlb\" (UniqueName: \"kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084154 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084154 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084342 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084342 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.084795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.084745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.086397 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.086379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.086621 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.086602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.093891 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.093872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rlb\" (UniqueName: \"kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.121656 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.121627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:32:00.125063 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.125048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.127820 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.127792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-qjzf2\""
Jul 16 18:32:00.133191 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.133169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:32:00.147268 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.147211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:00.209594 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.209575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:00.285664 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.285634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.285792 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.285695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.285792 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.285730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2rn\" (UniqueName: \"kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.341821 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.341798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:32:00.343420 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:32:00.343396    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f94afe_0bf4_4321_b690_d1994284ac3d.slice/crio-f7f15095eb46cf9826884813d1c466d8f1d95c9421188f3a0e2cd1f72f473c8f WatchSource:0}: Error finding container f7f15095eb46cf9826884813d1c466d8f1d95c9421188f3a0e2cd1f72f473c8f: Status 404 returned error can't find the container with id f7f15095eb46cf9826884813d1c466d8f1d95c9421188f3a0e2cd1f72f473c8f
Jul 16 18:32:00.386569 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.386544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.386666 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.386578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2rn\" (UniqueName: \"kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.386666 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.386644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.386956 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.386937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.388891 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.388867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.396133 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.396109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2rn\" (UniqueName: \"kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.435179 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.435115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:00.569827 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:00.569801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:32:00.571660 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:32:00.571634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd4be48_9102_4600_8473_cd7844b7ede4.slice/crio-33d02fbe802fd83a3aaf3f20788b352137b3f06cb1c55f58cf2665cfd02bbfee WatchSource:0}: Error finding container 33d02fbe802fd83a3aaf3f20788b352137b3f06cb1c55f58cf2665cfd02bbfee: Status 404 returned error can't find the container with id 33d02fbe802fd83a3aaf3f20788b352137b3f06cb1c55f58cf2665cfd02bbfee
Jul 16 18:32:01.211656 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.211619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" event={"ID":"74f94afe-0bf4-4321-b690-d1994284ac3d","Type":"ContainerStarted","Data":"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"}
Jul 16 18:32:01.212104 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.211677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" event={"ID":"74f94afe-0bf4-4321-b690-d1994284ac3d","Type":"ContainerStarted","Data":"f7f15095eb46cf9826884813d1c466d8f1d95c9421188f3a0e2cd1f72f473c8f"}
Jul 16 18:32:01.213118 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.213091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" event={"ID":"0fd4be48-9102-4600-8473-cd7844b7ede4","Type":"ContainerStarted","Data":"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"}
Jul 16 18:32:01.213210 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.213135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" event={"ID":"0fd4be48-9102-4600-8473-cd7844b7ede4","Type":"ContainerStarted","Data":"33d02fbe802fd83a3aaf3f20788b352137b3f06cb1c55f58cf2665cfd02bbfee"}
Jul 16 18:32:01.213210 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.213156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:01.232899 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.232856    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" podStartSLOduration=2.232843232 podStartE2EDuration="2.232843232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:32:01.232101129 +0000 UTC m=+3447.988509223" watchObservedRunningTime="2026-07-16 18:32:01.232843232 +0000 UTC m=+3447.989251294"
Jul 16 18:32:01.251099 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:01.251052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" podStartSLOduration=1.25103784 podStartE2EDuration="1.25103784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:32:01.250972087 +0000 UTC m=+3448.007380150" watchObservedRunningTime="2026-07-16 18:32:01.25103784 +0000 UTC m=+3448.007445903"
Jul 16 18:32:02.210400 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:02.210366    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:10.210473 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:10.210437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:10.222728 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:10.222704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:10.260777 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:10.260754    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:32:17.729426 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:17.729392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:32.219044 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:32.219014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:32:35.638995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:35.638949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:37.558924 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:37.558884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:46.640230 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:46.640193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:48.132782 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:48.132746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:32:48.144644 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:48.144620    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:32:48.156055 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:48.156027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:48.216568 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:48.216527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:32:52.218756 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:32:52.218720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:33:02.218101 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:02.218015    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:33:03.153188 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.153132    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerName="main" containerID="cri-o://c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691" gracePeriod=45
Jul 16 18:33:03.167254 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.167220    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main" containerID="cri-o://934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65" gracePeriod=45
Jul 16 18:33:03.250812 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.250779    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 16 18:33:03.423183 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.423160    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:33:03.433572 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.433550    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:33:03.463704 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.463671    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerID="c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691" exitCode=0
Jul 16 18:33:03.463861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.463743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"
Jul 16 18:33:03.463861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.463764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" event={"ID":"74f94afe-0bf4-4321-b690-d1994284ac3d","Type":"ContainerDied","Data":"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"}
Jul 16 18:33:03.463998 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.463875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b" event={"ID":"74f94afe-0bf4-4321-b690-d1994284ac3d","Type":"ContainerDied","Data":"f7f15095eb46cf9826884813d1c466d8f1d95c9421188f3a0e2cd1f72f473c8f"}
Jul 16 18:33:03.463998 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.463908    2577 scope.go:117] "RemoveContainer" containerID="c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"
Jul 16 18:33:03.465436 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.465410    2577 generic.go:356] "Generic (PLEG): container finished" podID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerID="934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65" exitCode=0
Jul 16 18:33:03.465574 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.465434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" event={"ID":"0fd4be48-9102-4600-8473-cd7844b7ede4","Type":"ContainerDied","Data":"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"}
Jul 16 18:33:03.465574 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.465469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"
Jul 16 18:33:03.465678 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.465473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8" event={"ID":"0fd4be48-9102-4600-8473-cd7844b7ede4","Type":"ContainerDied","Data":"33d02fbe802fd83a3aaf3f20788b352137b3f06cb1c55f58cf2665cfd02bbfee"}
Jul 16 18:33:03.471986 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.471965    2577 scope.go:117] "RemoveContainer" containerID="c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"
Jul 16 18:33:03.472276 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:33:03.472252    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691\": container with ID starting with c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691 not found: ID does not exist" containerID="c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"
Jul 16 18:33:03.472344 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.472284    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691"} err="failed to get container status \"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691\": rpc error: code = NotFound desc = could not find container \"c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691\": container with ID starting with c8322f4c13de6d89b4badc057d6aa65212bf75d13e94d6be19492b58e3728691 not found: ID does not exist"
Jul 16 18:33:03.472344 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.472305    2577 scope.go:117] "RemoveContainer" containerID="934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"
Jul 16 18:33:03.480456 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.480439    2577 scope.go:117] "RemoveContainer" containerID="934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"
Jul 16 18:33:03.480700 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:33:03.480683    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65\": container with ID starting with 934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65 not found: ID does not exist" containerID="934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"
Jul 16 18:33:03.480767 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.480707    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65"} err="failed to get container status \"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65\": rpc error: code = NotFound desc = could not find container \"934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65\": container with ID starting with 934ef95ec043e38268bfbde39999251c8cb0835281f3f10c4250eb0ff715ca65 not found: ID does not exist"
Jul 16 18:33:03.577293 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577293 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs\") pod \"0fd4be48-9102-4600-8473-cd7844b7ede4\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") "
Jul 16 18:33:03.577530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds\") pod \"0fd4be48-9102-4600-8473-cd7844b7ede4\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") "
Jul 16 18:33:03.577530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9rlb\" (UniqueName: \"kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577533    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm\") pod \"74f94afe-0bf4-4321-b690-d1994284ac3d\" (UID: \"74f94afe-0bf4-4321-b690-d1994284ac3d\") "
Jul 16 18:33:03.577795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc2rn\" (UniqueName: \"kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn\") pod \"0fd4be48-9102-4600-8473-cd7844b7ede4\" (UID: \"0fd4be48-9102-4600-8473-cd7844b7ede4\") "
Jul 16 18:33:03.577795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home" (OuterVolumeSpecName: "home") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:33:03.577795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0fd4be48-9102-4600-8473-cd7844b7ede4" (UID: "0fd4be48-9102-4600-8473-cd7844b7ede4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:33:03.577987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:33:03.577987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577865    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.577987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577885    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fd4be48-9102-4600-8473-cd7844b7ede4-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.577987 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.577864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:33:03.579719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.579690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm" (OuterVolumeSpecName: "dshm") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:33:03.579719 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.579705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn" (OuterVolumeSpecName: "kube-api-access-gc2rn") pod "0fd4be48-9102-4600-8473-cd7844b7ede4" (UID: "0fd4be48-9102-4600-8473-cd7844b7ede4"). InnerVolumeSpecName "kube-api-access-gc2rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:33:03.579883 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.579812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fd4be48-9102-4600-8473-cd7844b7ede4" (UID: "0fd4be48-9102-4600-8473-cd7844b7ede4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:33:03.579928 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.579906    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:33:03.580013 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.579989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb" (OuterVolumeSpecName: "kube-api-access-c9rlb") pod "74f94afe-0bf4-4321-b690-d1994284ac3d" (UID: "74f94afe-0bf4-4321-b690-d1994284ac3d"). InnerVolumeSpecName "kube-api-access-c9rlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678746    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rlb\" (UniqueName: \"kubernetes.io/projected/74f94afe-0bf4-4321-b690-d1994284ac3d-kube-api-access-c9rlb\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678776    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678789    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2rn\" (UniqueName: \"kubernetes.io/projected/0fd4be48-9102-4600-8473-cd7844b7ede4-kube-api-access-gc2rn\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678802    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f94afe-0bf4-4321-b690-d1994284ac3d-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678813    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd4be48-9102-4600-8473-cd7844b7ede4-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.678829 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678825    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.679103 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.678860    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f94afe-0bf4-4321-b690-d1994284ac3d-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:33:03.790929 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.790904    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:33:03.795882 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.795857    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-mmp8b"]
Jul 16 18:33:03.809469 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.809446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:33:03.814722 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:03.814698    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fc8nwn8"]
Jul 16 18:33:04.304044 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:04.304017    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" path="/var/lib/kubelet/pods/0fd4be48-9102-4600-8473-cd7844b7ede4/volumes"
Jul 16 18:33:04.304386 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:04.304378    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" path="/var/lib/kubelet/pods/74f94afe-0bf4-4321-b690-d1994284ac3d/volumes"
Jul 16 18:33:27.710634 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710603    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710895    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main"
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710906    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main"
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710920    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerName="main"
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710925    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerName="main"
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710971    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd4be48-9102-4600-8473-cd7844b7ede4" containerName="main"
Jul 16 18:33:27.710995 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.710984    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f94afe-0bf4-4321-b690-d1994284ac3d" containerName="main"
Jul 16 18:33:27.713682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.713666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.716366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.716341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 16 18:33:27.717525 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.717487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:33:27.725277 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.725255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:33:27.744287 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.744260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:33:27.760559 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.760643 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.760643 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.760716 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.760716 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v67w\" (UniqueName: \"kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.760782 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.760726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.861886 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.861861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.861980 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.861898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.861980 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.861916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v67w\" (UniqueName: \"kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.861980 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.861955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.862130 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.862021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.862130 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.862061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.862266 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.862243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.862326 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.862279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.862375 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.862360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.864210 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.864183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.864479 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.864457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.874231 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.874210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v67w\" (UniqueName: \"kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:27.999822 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:27.999789    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:33:28.003098 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.003083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.006317 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.006299    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-jdqvj\""
Jul 16 18:33:28.013865 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.013842    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:33:28.023467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.023448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:28.049699 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.049675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:33:28.063857 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.063833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.063990 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.063866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqfb\" (UniqueName: \"kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.063990 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.063913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.165014 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.164979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.165191 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.165069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.165191 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.165093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqfb\" (UniqueName: \"kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.165376 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.165354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.167942 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.167918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.169151 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.169131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:33:28.171111 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:33:28.171088    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c10cf6_c893_4c43_88af_3083590e7aec.slice/crio-0d0e3be530d5dad05bdd29dfe71c9b9bfe23d28b8062728fbd5777a364856029 WatchSource:0}: Error finding container 0d0e3be530d5dad05bdd29dfe71c9b9bfe23d28b8062728fbd5777a364856029: Status 404 returned error can't find the container with id 0d0e3be530d5dad05bdd29dfe71c9b9bfe23d28b8062728fbd5777a364856029
Jul 16 18:33:28.181446 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.181410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqfb\" (UniqueName: \"kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.315287 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.315223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.442728 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.442640    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:33:28.445165 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:33:28.445143    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02ea61_b9c6_485c_a4ae_e0f1c6409a78.slice/crio-c7fb941d3c0e5080a3f49fb9e3994102f243f16524d9ded18b6a04c271303aba WatchSource:0}: Error finding container c7fb941d3c0e5080a3f49fb9e3994102f243f16524d9ded18b6a04c271303aba: Status 404 returned error can't find the container with id c7fb941d3c0e5080a3f49fb9e3994102f243f16524d9ded18b6a04c271303aba
Jul 16 18:33:28.563217 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.563184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" event={"ID":"51c10cf6-c893-4c43-88af-3083590e7aec","Type":"ContainerStarted","Data":"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"}
Jul 16 18:33:28.563380 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.563240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" event={"ID":"51c10cf6-c893-4c43-88af-3083590e7aec","Type":"ContainerStarted","Data":"0d0e3be530d5dad05bdd29dfe71c9b9bfe23d28b8062728fbd5777a364856029"}
Jul 16 18:33:28.565046 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.565018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" event={"ID":"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78","Type":"ContainerStarted","Data":"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"}
Jul 16 18:33:28.565046 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.565059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" event={"ID":"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78","Type":"ContainerStarted","Data":"c7fb941d3c0e5080a3f49fb9e3994102f243f16524d9ded18b6a04c271303aba"}
Jul 16 18:33:28.565214 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.565104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:33:28.609994 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.609894    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" podStartSLOduration=1.609878389 podStartE2EDuration="1.609878389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:33:28.607682675 +0000 UTC m=+3535.364090736" watchObservedRunningTime="2026-07-16 18:33:28.609878389 +0000 UTC m=+3535.366286613"
Jul 16 18:33:28.644396 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:28.644343    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" podStartSLOduration=1.64432555 podStartE2EDuration="1.64432555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:33:28.644176567 +0000 UTC m=+3535.400584629" watchObservedRunningTime="2026-07-16 18:33:28.64432555 +0000 UTC m=+3535.400733618"
Jul 16 18:33:29.023639 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:29.023608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:38.024197 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:38.024159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:38.036735 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:38.036706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:38.617096 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:38.617065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:33:45.588163 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:45.588125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:33:48.142076 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:48.142040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:33:48.248320 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:48.248291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:33:59.572151 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:33:59.572121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:34:00.978769 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:00.978725    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:34:00.991444 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:00.991415    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:34:01.003147 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:01.003107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:01.103246 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:01.103215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:09.309386 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:09.309348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:09.572788 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:09.572710    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:34:16.007169 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.007125    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" podUID="51c10cf6-c893-4c43-88af-3083590e7aec" containerName="main" containerID="cri-o://e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990" gracePeriod=45
Jul 16 18:34:16.020663 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.020609    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerName="main" containerID="cri-o://95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4" gracePeriod=45
Jul 16 18:34:16.278092 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.278067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:34:16.280912 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.280892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:34:16.449085 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449225 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs\") pod \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") "
Jul 16 18:34:16.449225 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449225 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfqfb\" (UniqueName: \"kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb\") pod \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") "
Jul 16 18:34:16.449225 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v67w\" (UniqueName: \"kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds\") pod \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\" (UID: \"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78\") "
Jul 16 18:34:16.449467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm\") pod \"51c10cf6-c893-4c43-88af-3083590e7aec\" (UID: \"51c10cf6-c893-4c43-88af-3083590e7aec\") "
Jul 16 18:34:16.449467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache" (OuterVolumeSpecName: "model-cache") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:16.449773 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449612    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.449773 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" (UID: "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:16.449773 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:16.449773 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.449685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home" (OuterVolumeSpecName: "home") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:16.451373 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.451346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:34:16.451486 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.451370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb" (OuterVolumeSpecName: "kube-api-access-xfqfb") pod "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" (UID: "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78"). InnerVolumeSpecName "kube-api-access-xfqfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:34:16.451486 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.451450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm" (OuterVolumeSpecName: "dshm") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:16.451675 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.451649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w" (OuterVolumeSpecName: "kube-api-access-4v67w") pod "51c10cf6-c893-4c43-88af-3083590e7aec" (UID: "51c10cf6-c893-4c43-88af-3083590e7aec"). InnerVolumeSpecName "kube-api-access-4v67w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:34:16.451787 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.451695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" (UID: "8c02ea61-b9c6-485c-a4ae-e0f1c6409a78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:34:16.550900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550842    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v67w\" (UniqueName: \"kubernetes.io/projected/51c10cf6-c893-4c43-88af-3083590e7aec-kube-api-access-4v67w\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.550900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550865    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.550900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550894    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.550900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550901    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.551083 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550909    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.551083 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550917    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c10cf6-c893-4c43-88af-3083590e7aec-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.551083 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550925    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqfb\" (UniqueName: \"kubernetes.io/projected/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78-kube-api-access-xfqfb\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.551083 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.550932    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c10cf6-c893-4c43-88af-3083590e7aec-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:34:16.757973 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.757939    2577 generic.go:356] "Generic (PLEG): container finished" podID="51c10cf6-c893-4c43-88af-3083590e7aec" containerID="e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990" exitCode=0
Jul 16 18:34:16.758108 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.758007    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"
Jul 16 18:34:16.758108 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.758034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" event={"ID":"51c10cf6-c893-4c43-88af-3083590e7aec","Type":"ContainerDied","Data":"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"}
Jul 16 18:34:16.758108 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.758097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477" event={"ID":"51c10cf6-c893-4c43-88af-3083590e7aec","Type":"ContainerDied","Data":"0d0e3be530d5dad05bdd29dfe71c9b9bfe23d28b8062728fbd5777a364856029"}
Jul 16 18:34:16.758270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.758123    2577 scope.go:117] "RemoveContainer" containerID="e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"
Jul 16 18:34:16.759601 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.759572    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerID="95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4" exitCode=0
Jul 16 18:34:16.759723 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.759612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" event={"ID":"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78","Type":"ContainerDied","Data":"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"}
Jul 16 18:34:16.759723 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.759638    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"
Jul 16 18:34:16.759723 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.759664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d" event={"ID":"8c02ea61-b9c6-485c-a4ae-e0f1c6409a78","Type":"ContainerDied","Data":"c7fb941d3c0e5080a3f49fb9e3994102f243f16524d9ded18b6a04c271303aba"}
Jul 16 18:34:16.768414 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.768386    2577 scope.go:117] "RemoveContainer" containerID="e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"
Jul 16 18:34:16.768751 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:34:16.768727    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990\": container with ID starting with e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990 not found: ID does not exist" containerID="e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"
Jul 16 18:34:16.768828 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.768762    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990"} err="failed to get container status \"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990\": rpc error: code = NotFound desc = could not find container \"e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990\": container with ID starting with e1327515703d4850320cd0ff0328bf0dd3481b16d808459acd254c6cc70e6990 not found: ID does not exist"
Jul 16 18:34:16.768828 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.768788    2577 scope.go:117] "RemoveContainer" containerID="95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"
Jul 16 18:34:16.776627 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.776607    2577 scope.go:117] "RemoveContainer" containerID="95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"
Jul 16 18:34:16.776890 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:34:16.776871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4\": container with ID starting with 95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4 not found: ID does not exist" containerID="95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"
Jul 16 18:34:16.776932 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.776897    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4"} err="failed to get container status \"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4\": rpc error: code = NotFound desc = could not find container \"95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4\": container with ID starting with 95332afbe09f6ee081a24bdb3f1e161459fbbd5a2e08081e6c3272c105ebe3b4 not found: ID does not exist"
Jul 16 18:34:16.787880 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.787854    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:34:16.793558 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.793534    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4rn477"]
Jul 16 18:34:16.807768 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.807713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:34:16.815338 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:16.815318    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schev5h4d"]
Jul 16 18:34:18.304290 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:18.304258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c10cf6-c893-4c43-88af-3083590e7aec" path="/var/lib/kubelet/pods/51c10cf6-c893-4c43-88af-3083590e7aec/volumes"
Jul 16 18:34:18.304661 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:18.304631    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" path="/var/lib/kubelet/pods/8c02ea61-b9c6-485c-a4ae-e0f1c6409a78/volumes"
Jul 16 18:34:25.609103 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609029    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609306    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerName="main"
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609317    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerName="main"
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c10cf6-c893-4c43-88af-3083590e7aec" containerName="main"
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c10cf6-c893-4c43-88af-3083590e7aec" containerName="main"
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609393    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c10cf6-c893-4c43-88af-3083590e7aec" containerName="main"
Jul 16 18:34:25.609440 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.609401    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c02ea61-b9c6-485c-a4ae-e0f1c6409a78" containerName="main"
Jul 16 18:34:25.612566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.612549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.616164 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.616145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 16 18:34:25.616259 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.616205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:34:25.625981 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.625957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:34:25.645768 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.645738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:25.724173 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.724265 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.724265 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4j4\" (UniqueName: \"kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.724355 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.724355 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.724355 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.724349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825690 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4j4\" (UniqueName: \"kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825794 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825907 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.825907 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.825857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.826141 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.826123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.826195 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.826174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.826234 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.826186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.828054 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.828031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.828303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.828286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.835714 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.835695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4j4\" (UniqueName: \"kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.922435 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.922373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:25.937858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.937679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:34:25.941509 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.941482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:25.944125 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.944105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-t5tw7\""
Jul 16 18:34:25.952630 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.952607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:34:25.969999 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:25.969958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:26.063310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.063281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:34:26.065769 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:34:26.065743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459bcfab_bf82_473a_ac64_6ab71be3e193.slice/crio-b217c3ad4c75a7162cf08271e891f6e769ace0b121e2e8b079221823901dd291 WatchSource:0}: Error finding container b217c3ad4c75a7162cf08271e891f6e769ace0b121e2e8b079221823901dd291: Status 404 returned error can't find the container with id b217c3ad4c75a7162cf08271e891f6e769ace0b121e2e8b079221823901dd291
Jul 16 18:34:26.127818 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.127797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.127924 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.127863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbd7s\" (UniqueName: \"kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.127997 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.127971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.228981 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.228909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbd7s\" (UniqueName: \"kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.228981 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.228961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.229139 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.229000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.229402 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.229379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.231376 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.231359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.239702 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.239684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbd7s\" (UniqueName: \"kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.254547 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.254524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.390272 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.390243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:34:26.391696 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:34:26.391666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15447a4_ef93_4d6c_bce5_659b7243d2ad.slice/crio-88c95c4f0a17f6ad7c3d467a376d8653c6e42b3c0f226e70dd3dd971ac7ae4d9 WatchSource:0}: Error finding container 88c95c4f0a17f6ad7c3d467a376d8653c6e42b3c0f226e70dd3dd971ac7ae4d9: Status 404 returned error can't find the container with id 88c95c4f0a17f6ad7c3d467a376d8653c6e42b3c0f226e70dd3dd971ac7ae4d9
Jul 16 18:34:26.804766 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.804731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" event={"ID":"b15447a4-ef93-4d6c-bce5-659b7243d2ad","Type":"ContainerStarted","Data":"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"}
Jul 16 18:34:26.805206 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.804878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:34:26.805206 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.804926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" event={"ID":"b15447a4-ef93-4d6c-bce5-659b7243d2ad","Type":"ContainerStarted","Data":"88c95c4f0a17f6ad7c3d467a376d8653c6e42b3c0f226e70dd3dd971ac7ae4d9"}
Jul 16 18:34:26.806234 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.806203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" event={"ID":"459bcfab-bf82-473a-ac64-6ab71be3e193","Type":"ContainerStarted","Data":"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"}
Jul 16 18:34:26.806343 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.806244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" event={"ID":"459bcfab-bf82-473a-ac64-6ab71be3e193","Type":"ContainerStarted","Data":"b217c3ad4c75a7162cf08271e891f6e769ace0b121e2e8b079221823901dd291"}
Jul 16 18:34:26.834261 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.834207    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" podStartSLOduration=1.8341890730000001 podStartE2EDuration="1.834189073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:34:26.83294017 +0000 UTC m=+3593.589348233" watchObservedRunningTime="2026-07-16 18:34:26.834189073 +0000 UTC m=+3593.590597137"
Jul 16 18:34:26.853870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.853828    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" podStartSLOduration=1.853816347 podStartE2EDuration="1.853816347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:34:26.853531166 +0000 UTC m=+3593.609939228" watchObservedRunningTime="2026-07-16 18:34:26.853816347 +0000 UTC m=+3593.610224408"
Jul 16 18:34:26.922679 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:26.922649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:35.922918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:35.922885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:35.934964 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:35.934939    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:36.789027 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:36.788989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:34:36.857389 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:36.857363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:34:57.812965 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:34:57.812933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:35:18.187589 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:18.187520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:18.249615 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:18.249585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:27.150125 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:27.150090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:29.324360 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:29.324324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:29.333533 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:29.333489    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:35:29.341914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:29.341888    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:35:29.392193 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:29.391225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:37.812900 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:37.812864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:35:44.360560 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.360476    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerName="main" containerID="cri-o://6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611" gracePeriod=45
Jul 16 18:35:44.365532 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.365484    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" podUID="459bcfab-bf82-473a-ac64-6ab71be3e193" containerName="main" containerID="cri-o://cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c" gracePeriod=45
Jul 16 18:35:44.624523 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.624489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:35:44.627241 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.627221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:35:44.752475 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs\") pod \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") "
Jul 16 18:35:44.752612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752612 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbd7s\" (UniqueName: \"kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s\") pod \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") "
Jul 16 18:35:44.752780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752764    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c4j4\" (UniqueName: \"kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4\") pod \"459bcfab-bf82-473a-ac64-6ab71be3e193\" (UID: \"459bcfab-bf82-473a-ac64-6ab71be3e193\") "
Jul 16 18:35:44.752938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds\") pod \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\" (UID: \"b15447a4-ef93-4d6c-bce5-659b7243d2ad\") "
Jul 16 18:35:44.752938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home" (OuterVolumeSpecName: "home") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:35:44.752938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.752888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache" (OuterVolumeSpecName: "model-cache") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:35:44.753207 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b15447a4-ef93-4d6c-bce5-659b7243d2ad" (UID: "b15447a4-ef93-4d6c-bce5-659b7243d2ad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:35:44.753207 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:35:44.753403 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753384    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.753485 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753411    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.753485 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753430    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.753485 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.753443    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.754743 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.754718    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm" (OuterVolumeSpecName: "dshm") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:35:44.754835 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.754724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:35:44.754877 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.754840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s" (OuterVolumeSpecName: "kube-api-access-xbd7s") pod "b15447a4-ef93-4d6c-bce5-659b7243d2ad" (UID: "b15447a4-ef93-4d6c-bce5-659b7243d2ad"). InnerVolumeSpecName "kube-api-access-xbd7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:35:44.754991 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.754971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b15447a4-ef93-4d6c-bce5-659b7243d2ad" (UID: "b15447a4-ef93-4d6c-bce5-659b7243d2ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:35:44.755125 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.755109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4" (OuterVolumeSpecName: "kube-api-access-4c4j4") pod "459bcfab-bf82-473a-ac64-6ab71be3e193" (UID: "459bcfab-bf82-473a-ac64-6ab71be3e193"). InnerVolumeSpecName "kube-api-access-4c4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:35:44.854739 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.854717    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c4j4\" (UniqueName: \"kubernetes.io/projected/459bcfab-bf82-473a-ac64-6ab71be3e193-kube-api-access-4c4j4\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.854739 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.854739    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b15447a4-ef93-4d6c-bce5-659b7243d2ad-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.854858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.854749    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bcfab-bf82-473a-ac64-6ab71be3e193-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.854858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.854758    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bcfab-bf82-473a-ac64-6ab71be3e193-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:44.854858 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:44.854766    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbd7s\" (UniqueName: \"kubernetes.io/projected/b15447a4-ef93-4d6c-bce5-659b7243d2ad-kube-api-access-xbd7s\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:35:45.120411 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.120384    2577 generic.go:356] "Generic (PLEG): container finished" podID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerID="6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611" exitCode=0
Jul 16 18:35:45.120558 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.120426    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"
Jul 16 18:35:45.120558 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.120473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" event={"ID":"b15447a4-ef93-4d6c-bce5-659b7243d2ad","Type":"ContainerDied","Data":"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"}
Jul 16 18:35:45.120558 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.120551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v" event={"ID":"b15447a4-ef93-4d6c-bce5-659b7243d2ad","Type":"ContainerDied","Data":"88c95c4f0a17f6ad7c3d467a376d8653c6e42b3c0f226e70dd3dd971ac7ae4d9"}
Jul 16 18:35:45.120734 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.120582    2577 scope.go:117] "RemoveContainer" containerID="6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"
Jul 16 18:35:45.121805 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.121781    2577 generic.go:356] "Generic (PLEG): container finished" podID="459bcfab-bf82-473a-ac64-6ab71be3e193" containerID="cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c" exitCode=0
Jul 16 18:35:45.121915 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.121830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"
Jul 16 18:35:45.121915 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.121851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" event={"ID":"459bcfab-bf82-473a-ac64-6ab71be3e193","Type":"ContainerDied","Data":"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"}
Jul 16 18:35:45.121915 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.121891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h" event={"ID":"459bcfab-bf82-473a-ac64-6ab71be3e193","Type":"ContainerDied","Data":"b217c3ad4c75a7162cf08271e891f6e769ace0b121e2e8b079221823901dd291"}
Jul 16 18:35:45.130131 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.130109    2577 scope.go:117] "RemoveContainer" containerID="6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"
Jul 16 18:35:45.130409 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:35:45.130386    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611\": container with ID starting with 6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611 not found: ID does not exist" containerID="6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"
Jul 16 18:35:45.130655 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.130419    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611"} err="failed to get container status \"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611\": rpc error: code = NotFound desc = could not find container \"6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611\": container with ID starting with 6c6546bd41afbc1b7771f79916ae9c33d543c53cc702ea5664cfdd02e4920611 not found: ID does not exist"
Jul 16 18:35:45.130655 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.130436    2577 scope.go:117] "RemoveContainer" containerID="cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"
Jul 16 18:35:45.137852 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.137835    2577 scope.go:117] "RemoveContainer" containerID="cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"
Jul 16 18:35:45.138073 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:35:45.138055    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c\": container with ID starting with cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c not found: ID does not exist" containerID="cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"
Jul 16 18:35:45.138127 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.138080    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c"} err="failed to get container status \"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c\": rpc error: code = NotFound desc = could not find container \"cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c\": container with ID starting with cdd7d5e7d91012ae7e22f1a9b4a4900ca61cef433e38f8b50f4983368787a39c not found: ID does not exist"
Jul 16 18:35:45.148599 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.148578    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:35:45.151592 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.151574    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d7l7h"]
Jul 16 18:35:45.165309 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.165290    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:35:45.171803 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:45.171785    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffrln5v"]
Jul 16 18:35:46.304543 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:46.304485    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459bcfab-bf82-473a-ac64-6ab71be3e193" path="/var/lib/kubelet/pods/459bcfab-bf82-473a-ac64-6ab71be3e193/volumes"
Jul 16 18:35:46.304918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:46.304908    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" path="/var/lib/kubelet/pods/b15447a4-ef93-4d6c-bce5-659b7243d2ad/volumes"
Jul 16 18:35:50.514585 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514835    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459bcfab-bf82-473a-ac64-6ab71be3e193" containerName="main"
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514846    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="459bcfab-bf82-473a-ac64-6ab71be3e193" containerName="main"
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514858    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerName="main"
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514863    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerName="main"
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514920    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b15447a4-ef93-4d6c-bce5-659b7243d2ad" containerName="main"
Jul 16 18:35:50.515028 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.514928    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="459bcfab-bf82-473a-ac64-6ab71be3e193" containerName="main"
Jul 16 18:35:50.517801 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.517785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.520658 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.520625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:35:50.521602 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.521585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 16 18:35:50.530437 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.530417    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:35:50.570227 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.570194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:50.600000 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.599971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.600131 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.600008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xq8k\" (UniqueName: \"kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.600131 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.600035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.600131 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.600084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.600131 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.600124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.600273 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.600215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701464 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xq8k\" (UniqueName: \"kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.701959 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.702020 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.701972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.702020 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.702015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.703783 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.703759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.704035 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.704019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.711603 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.711582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xq8k\" (UniqueName: \"kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.828094 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.828003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:50.858954 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.858922    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:35:50.862285 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.862268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:50.865406 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.865384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-q68bc\""
Jul 16 18:35:50.874266 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.874241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:35:50.878861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.878831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:35:50.964629 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:50.964605    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:35:50.966912 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:35:50.966887    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a5d284_775b_455e_9a93_40f902a18ba2.slice/crio-6c67e7cbfc667fd39582632a80a8eece48daea376439da8068bbdf014801f307 WatchSource:0}: Error finding container 6c67e7cbfc667fd39582632a80a8eece48daea376439da8068bbdf014801f307: Status 404 returned error can't find the container with id 6c67e7cbfc667fd39582632a80a8eece48daea376439da8068bbdf014801f307
Jul 16 18:35:51.004557 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.004469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.004557 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.004519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26mm\" (UniqueName: \"kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.004557 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.004546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.004780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.004591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.004780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.004662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.105918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.105841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.105918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.105884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.105918 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.105911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.106165 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.106030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26mm\" (UniqueName: \"kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.106165 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.106073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.106295 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.106277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.106361 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.106333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.106418 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.106378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.108363 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.108346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.115580 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.115558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26mm\" (UniqueName: \"kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm\") pod \"scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.157353 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.157329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" event={"ID":"79a5d284-775b-455e-9a93-40f902a18ba2","Type":"ContainerStarted","Data":"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"}
Jul 16 18:35:51.157444 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.157371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" event={"ID":"79a5d284-775b-455e-9a93-40f902a18ba2","Type":"ContainerStarted","Data":"6c67e7cbfc667fd39582632a80a8eece48daea376439da8068bbdf014801f307"}
Jul 16 18:35:51.174566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.174543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:51.184072 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.184017    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" podStartSLOduration=1.183994549 podStartE2EDuration="1.183994549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:35:51.180215878 +0000 UTC m=+3677.936623939" watchObservedRunningTime="2026-07-16 18:35:51.183994549 +0000 UTC m=+3677.940402614"
Jul 16 18:35:51.302428 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.302403    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:35:51.304896 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:35:51.304872    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89c555f_9b0a_4604_9957_676879bb0c37.slice/crio-0fef9e32ffd85066c2da58868dcb30f5954d80c73c81e7896ef525c0d7d8fea6 WatchSource:0}: Error finding container 0fef9e32ffd85066c2da58868dcb30f5954d80c73c81e7896ef525c0d7d8fea6: Status 404 returned error can't find the container with id 0fef9e32ffd85066c2da58868dcb30f5954d80c73c81e7896ef525c0d7d8fea6
Jul 16 18:35:51.828559 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:51.828530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:35:52.162990 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:52.162902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" event={"ID":"c89c555f-9b0a-4604-9957-676879bb0c37","Type":"ContainerStarted","Data":"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"}
Jul 16 18:35:52.162990 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:52.162983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:35:52.163175 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:52.163019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" event={"ID":"c89c555f-9b0a-4604-9957-676879bb0c37","Type":"ContainerStarted","Data":"0fef9e32ffd85066c2da58868dcb30f5954d80c73c81e7896ef525c0d7d8fea6"}
Jul 16 18:35:52.186074 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:52.186013    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" podStartSLOduration=2.185999088 podStartE2EDuration="2.185999088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:35:52.184665125 +0000 UTC m=+3678.941073198" watchObservedRunningTime="2026-07-16 18:35:52.185999088 +0000 UTC m=+3678.942407150"
Jul 16 18:35:53.376870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:35:53.376842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:36:00.829163 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:00.829046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:36:00.841143 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:00.841125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:36:01.211711 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:01.211620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:36:02.312977 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.312939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:36:02.565022 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.564934    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:36:02.568348 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.568330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.573764 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.573743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-gjtst\""
Jul 16 18:36:02.573906 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.573810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 16 18:36:02.582530 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.582492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:36:02.631080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.631053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:36:02.694549 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.694527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9n7\" (UniqueName: \"kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.694661 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.694584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.694705 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.694654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.795980 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.795955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.796093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.795995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.796093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.796038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9n7\" (UniqueName: \"kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.796469 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.796450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.798342 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.798325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.806588 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.806568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9n7\" (UniqueName: \"kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:02.878163 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:02.878104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:03.012227 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:03.012201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:36:03.014092 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:36:03.014062    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65093ff3_0be7_49d3_8837_e040626a036a.slice/crio-e9430140d24717cbdb461109a460b69394f6ae0aca289ca175f18ecf3c6c0151 WatchSource:0}: Error finding container e9430140d24717cbdb461109a460b69394f6ae0aca289ca175f18ecf3c6c0151: Status 404 returned error can't find the container with id e9430140d24717cbdb461109a460b69394f6ae0aca289ca175f18ecf3c6c0151
Jul 16 18:36:03.212701 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:03.212604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" event={"ID":"65093ff3-0be7-49d3-8837-e040626a036a","Type":"ContainerStarted","Data":"5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b"}
Jul 16 18:36:03.212701 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:03.212670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" event={"ID":"65093ff3-0be7-49d3-8837-e040626a036a","Type":"ContainerStarted","Data":"e9430140d24717cbdb461109a460b69394f6ae0aca289ca175f18ecf3c6c0151"}
Jul 16 18:36:03.213209 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:03.213180    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:03.233365 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:03.233317    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" podStartSLOduration=1.23330024 podStartE2EDuration="1.23330024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:36:03.231431952 +0000 UTC m=+3689.987840015" watchObservedRunningTime="2026-07-16 18:36:03.23330024 +0000 UTC m=+3689.989708302"
Jul 16 18:36:23.168969 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:23.168930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:36:35.222521 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:35.222471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:36:43.420578 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:36:43.420541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:02.568159 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:02.568128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:04.101309 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:04.101277    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:37:04.101991 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:04.101935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:04.108518 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:04.108482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:37:04.173083 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:04.173048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:05.358234 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:05.358200    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:37:05.380153 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:05.380114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:05.477032 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:05.477001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:13.169914 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:13.169866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:15.222593 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.222555    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:15.445907 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.445867    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 16 18:37:15.446018 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:37:15.445910    2577 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main" containerID="cri-o://5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b"
Jul 16 18:37:15.446018 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.445971    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main" containerID="cri-o://5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b" gracePeriod=50
Jul 16 18:37:15.518052 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.518029    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6_65093ff3-0be7-49d3-8837-e040626a036a/main/0.log"
Jul 16 18:37:15.518196 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.518073    2577 generic.go:356] "Generic (PLEG): container finished" podID="65093ff3-0be7-49d3-8837-e040626a036a" containerID="5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b" exitCode=1
Jul 16 18:37:15.518196 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.518144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" event={"ID":"65093ff3-0be7-49d3-8837-e040626a036a","Type":"ContainerDied","Data":"5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b"}
Jul 16 18:37:15.583352 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.583326    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6_65093ff3-0be7-49d3-8837-e040626a036a/main/0.log"
Jul 16 18:37:15.585668 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.585651    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:37:15.681749 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.681725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk9n7\" (UniqueName: \"kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7\") pod \"65093ff3-0be7-49d3-8837-e040626a036a\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") "
Jul 16 18:37:15.681849 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.681761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds\") pod \"65093ff3-0be7-49d3-8837-e040626a036a\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") "
Jul 16 18:37:15.681849 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.681821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs\") pod \"65093ff3-0be7-49d3-8837-e040626a036a\" (UID: \"65093ff3-0be7-49d3-8837-e040626a036a\") "
Jul 16 18:37:15.682111 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.682084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "65093ff3-0be7-49d3-8837-e040626a036a" (UID: "65093ff3-0be7-49d3-8837-e040626a036a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:15.683751 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.683731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65093ff3-0be7-49d3-8837-e040626a036a" (UID: "65093ff3-0be7-49d3-8837-e040626a036a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:37:15.683864 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.683850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7" (OuterVolumeSpecName: "kube-api-access-tk9n7") pod "65093ff3-0be7-49d3-8837-e040626a036a" (UID: "65093ff3-0be7-49d3-8837-e040626a036a"). InnerVolumeSpecName "kube-api-access-tk9n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:37:15.783439 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.783386    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9n7\" (UniqueName: \"kubernetes.io/projected/65093ff3-0be7-49d3-8837-e040626a036a-kube-api-access-tk9n7\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:15.783439 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.783406    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65093ff3-0be7-49d3-8837-e040626a036a-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:15.783439 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:15.783415    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65093ff3-0be7-49d3-8837-e040626a036a-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:16.523574 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.523548    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6_65093ff3-0be7-49d3-8837-e040626a036a/main/0.log"
Jul 16 18:37:16.523985 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.523651    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"
Jul 16 18:37:16.523985 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.523680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6" event={"ID":"65093ff3-0be7-49d3-8837-e040626a036a","Type":"ContainerDied","Data":"e9430140d24717cbdb461109a460b69394f6ae0aca289ca175f18ecf3c6c0151"}
Jul 16 18:37:16.523985 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.523734    2577 scope.go:117] "RemoveContainer" containerID="5b9e68dec664694c1badd35fc2034e27e4049378b6c2b30adc44c7e6d06de37b"
Jul 16 18:37:16.545512 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.545485    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:37:16.550826 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:16.550807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85ft6fh6"]
Jul 16 18:37:18.304432 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:18.304397    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65093ff3-0be7-49d3-8837-e040626a036a" path="/var/lib/kubelet/pods/65093ff3-0be7-49d3-8837-e040626a036a/volumes"
Jul 16 18:37:19.121936 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.121876    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" containerName="main" containerID="cri-o://23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10" gracePeriod=45
Jul 16 18:37:19.127346 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.127314    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" containerName="main" containerID="cri-o://2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac" gracePeriod=45
Jul 16 18:37:19.200828 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.200790    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 16 18:37:19.364813 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.364787    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:37:19.391649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.391544    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:37:19.514608 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xq8k\" (UniqueName: \"kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache\") pod \"c89c555f-9b0a-4604-9957-676879bb0c37\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs\") pod \"c89c555f-9b0a-4604-9957-676879bb0c37\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") "
Jul 16 18:37:19.514802 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds\") pod \"c89c555f-9b0a-4604-9957-676879bb0c37\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") "
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.514923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c89c555f-9b0a-4604-9957-676879bb0c37" (UID: "c89c555f-9b0a-4604-9957-676879bb0c37"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "c89c555f-9b0a-4604-9957-676879bb0c37" (UID: "c89c555f-9b0a-4604-9957-676879bb0c37"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home" (OuterVolumeSpecName: "home") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache\") pod \"c89c555f-9b0a-4604-9957-676879bb0c37\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") "
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir\") pod \"79a5d284-775b-455e-9a93-40f902a18ba2\" (UID: \"79a5d284-775b-455e-9a93-40f902a18ba2\") "
Jul 16 18:37:19.515184 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r26mm\" (UniqueName: \"kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm\") pod \"c89c555f-9b0a-4604-9957-676879bb0c37\" (UID: \"c89c555f-9b0a-4604-9957-676879bb0c37\") "
Jul 16 18:37:19.515577 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "c89c555f-9b0a-4604-9957-676879bb0c37" (UID: "c89c555f-9b0a-4604-9957-676879bb0c37"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515577 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.515577 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515562    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.515757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515581    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.515757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515596    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.515757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515610    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.515757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515622    2577 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-cachi2-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.515757 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.515636    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c89c555f-9b0a-4604-9957-676879bb0c37-tokenizer-uds\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.516949 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.516924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm" (OuterVolumeSpecName: "dshm") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:19.517063 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.516924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k" (OuterVolumeSpecName: "kube-api-access-9xq8k") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "kube-api-access-9xq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:37:19.517152 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.517136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm" (OuterVolumeSpecName: "kube-api-access-r26mm") pod "c89c555f-9b0a-4604-9957-676879bb0c37" (UID: "c89c555f-9b0a-4604-9957-676879bb0c37"). InnerVolumeSpecName "kube-api-access-r26mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:37:19.517243 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.517225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79a5d284-775b-455e-9a93-40f902a18ba2" (UID: "79a5d284-775b-455e-9a93-40f902a18ba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:37:19.517282 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.517241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c89c555f-9b0a-4604-9957-676879bb0c37" (UID: "c89c555f-9b0a-4604-9957-676879bb0c37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:37:19.537973 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.537952    2577 generic.go:356] "Generic (PLEG): container finished" podID="79a5d284-775b-455e-9a93-40f902a18ba2" containerID="23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10" exitCode=0
Jul 16 18:37:19.538066 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.537996    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"
Jul 16 18:37:19.538066 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.538040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" event={"ID":"79a5d284-775b-455e-9a93-40f902a18ba2","Type":"ContainerDied","Data":"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"}
Jul 16 18:37:19.538149 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.538085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r" event={"ID":"79a5d284-775b-455e-9a93-40f902a18ba2","Type":"ContainerDied","Data":"6c67e7cbfc667fd39582632a80a8eece48daea376439da8068bbdf014801f307"}
Jul 16 18:37:19.538149 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.538116    2577 scope.go:117] "RemoveContainer" containerID="23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"
Jul 16 18:37:19.539283 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.539266    2577 generic.go:356] "Generic (PLEG): container finished" podID="c89c555f-9b0a-4604-9957-676879bb0c37" containerID="2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac" exitCode=0
Jul 16 18:37:19.539377 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.539293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" event={"ID":"c89c555f-9b0a-4604-9957-676879bb0c37","Type":"ContainerDied","Data":"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"}
Jul 16 18:37:19.539377 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.539312    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"
Jul 16 18:37:19.539377 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.539323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2" event={"ID":"c89c555f-9b0a-4604-9957-676879bb0c37","Type":"ContainerDied","Data":"0fef9e32ffd85066c2da58868dcb30f5954d80c73c81e7896ef525c0d7d8fea6"}
Jul 16 18:37:19.547182 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.547161    2577 scope.go:117] "RemoveContainer" containerID="23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"
Jul 16 18:37:19.547532 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:37:19.547494    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10\": container with ID starting with 23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10 not found: ID does not exist" containerID="23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"
Jul 16 18:37:19.547600 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.547543    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10"} err="failed to get container status \"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10\": rpc error: code = NotFound desc = could not find container \"23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10\": container with ID starting with 23e2599ecb5882e053e2ea7c0c8d5fdf56486652ca727a448aaddc4cc6128b10 not found: ID does not exist"
Jul 16 18:37:19.547600 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.547564    2577 scope.go:117] "RemoveContainer" containerID="2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"
Jul 16 18:37:19.554994 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.554978    2577 scope.go:117] "RemoveContainer" containerID="2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"
Jul 16 18:37:19.555233 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:37:19.555216    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac\": container with ID starting with 2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac not found: ID does not exist" containerID="2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"
Jul 16 18:37:19.555282 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.555238    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac"} err="failed to get container status \"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac\": rpc error: code = NotFound desc = could not find container \"2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac\": container with ID starting with 2476194925bc46e396eca8d9bb60f6498ce7799826455b37b4d035ac93b6e3ac not found: ID does not exist"
Jul 16 18:37:19.564905 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.564886    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:37:19.570782 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.570765    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-bgg6r"]
Jul 16 18:37:19.584408 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.584390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:37:19.589255 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.589237    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dmbzl2"]
Jul 16 18:37:19.616974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.616956    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a5d284-775b-455e-9a93-40f902a18ba2-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.616974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.616974    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xq8k\" (UniqueName: \"kubernetes.io/projected/79a5d284-775b-455e-9a93-40f902a18ba2-kube-api-access-9xq8k\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.617093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.616983    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a5d284-775b-455e-9a93-40f902a18ba2-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.617093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.616994    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89c555f-9b0a-4604-9957-676879bb0c37-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:19.617093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:19.617002    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26mm\" (UniqueName: \"kubernetes.io/projected/c89c555f-9b0a-4604-9957-676879bb0c37-kube-api-access-r26mm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:37:20.304396 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:20.304362    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" path="/var/lib/kubelet/pods/79a5d284-775b-455e-9a93-40f902a18ba2/volumes"
Jul 16 18:37:20.304795 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:20.304781    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" path="/var/lib/kubelet/pods/c89c555f-9b0a-4604-9957-676879bb0c37/volumes"
Jul 16 18:37:23.560190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"]
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560420    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560430    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560444    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560460    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560465    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560544    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a5d284-775b-455e-9a93-40f902a18ba2" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65093ff3-0be7-49d3-8837-e040626a036a" containerName="main"
Jul 16 18:37:23.560695 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.560558    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89c555f-9b0a-4604-9957-676879bb0c37" containerName="main"
Jul 16 18:37:23.564790 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.564770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.568423 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.568399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:37:23.568575 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.568537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 16 18:37:23.579669 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.579645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"]
Jul 16 18:37:23.619609 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.619577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:23.649067 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx27c\" (UniqueName: \"kubernetes.io/projected/83cb5708-7fed-4211-9449-958f45154f9c-kube-api-access-wx27c\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.649167 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.649167 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.649167 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.649271 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.649271 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.649242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83cb5708-7fed-4211-9449-958f45154f9c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.749846 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx27c\" (UniqueName: \"kubernetes.io/projected/83cb5708-7fed-4211-9449-958f45154f9c-kube-api-access-wx27c\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.749938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.749938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.749938 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.750085 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.750085 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.749983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83cb5708-7fed-4211-9449-958f45154f9c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.750271 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.750252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.750339 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.750298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.750339 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.750333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.752253 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.752232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83cb5708-7fed-4211-9449-958f45154f9c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.752394 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.752376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83cb5708-7fed-4211-9449-958f45154f9c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.760686 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.760666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx27c\" (UniqueName: \"kubernetes.io/projected/83cb5708-7fed-4211-9449-958f45154f9c-kube-api-access-wx27c\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt\" (UID: \"83cb5708-7fed-4211-9449-958f45154f9c\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.860021 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.859968    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"]
Jul 16 18:37:23.863200 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.863186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:23.866093 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.866075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-nfl29\""
Jul 16 18:37:23.871889 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.871866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"]
Jul 16 18:37:23.876202 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.876180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:23.884700 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.884672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:23.951319 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.951292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:23.951458 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.951337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:23.951458 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:23.951365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqq7z\" (UniqueName: \"kubernetes.io/projected/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-kube-api-access-zqq7z\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.023146 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.023111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"]
Jul 16 18:37:24.026879 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:37:24.026855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cb5708_7fed_4211_9449_958f45154f9c.slice/crio-55e2251d6ac47a8f4ef5fa9a0440dff0d9f61098632dd9715c5d2f439a1ea45f WatchSource:0}: Error finding container 55e2251d6ac47a8f4ef5fa9a0440dff0d9f61098632dd9715c5d2f439a1ea45f: Status 404 returned error can't find the container with id 55e2251d6ac47a8f4ef5fa9a0440dff0d9f61098632dd9715c5d2f439a1ea45f
Jul 16 18:37:24.052670 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.052645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.052774 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.052693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.052774 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.052728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqq7z\" (UniqueName: \"kubernetes.io/projected/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-kube-api-access-zqq7z\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.053079 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.053059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.054763 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.054747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.080295 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.080271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqq7z\" (UniqueName: \"kubernetes.io/projected/0683c982-6d4c-4bd1-aa07-1f6be8ea3e55-kube-api-access-zqq7z\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k\" (UID: \"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.173144 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.173076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.294422 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.294397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"]
Jul 16 18:37:24.296398 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:37:24.296367    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683c982_6d4c_4bd1_aa07_1f6be8ea3e55.slice/crio-4acfd46ddfbb26c8d90fc266ef339105989c958cc240b3b70fc1b4fd416e752c WatchSource:0}: Error finding container 4acfd46ddfbb26c8d90fc266ef339105989c958cc240b3b70fc1b4fd416e752c: Status 404 returned error can't find the container with id 4acfd46ddfbb26c8d90fc266ef339105989c958cc240b3b70fc1b4fd416e752c
Jul 16 18:37:24.563422 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.563388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt" event={"ID":"83cb5708-7fed-4211-9449-958f45154f9c","Type":"ContainerStarted","Data":"6203a48e37284b2f6bd47774af61011d98949ca8d84a0dcf8d8b44e3eca8d030"}
Jul 16 18:37:24.563824 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.563445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt" event={"ID":"83cb5708-7fed-4211-9449-958f45154f9c","Type":"ContainerStarted","Data":"55e2251d6ac47a8f4ef5fa9a0440dff0d9f61098632dd9715c5d2f439a1ea45f"}
Jul 16 18:37:24.564916 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.564895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k" event={"ID":"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55","Type":"ContainerStarted","Data":"2d1bed46a7aed23f2f9c20c3e508fe36cd9855b06f1b12f1c13956686c473790"}
Jul 16 18:37:24.565033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.564930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k" event={"ID":"0683c982-6d4c-4bd1-aa07-1f6be8ea3e55","Type":"ContainerStarted","Data":"4acfd46ddfbb26c8d90fc266ef339105989c958cc240b3b70fc1b4fd416e752c"}
Jul 16 18:37:24.565033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.564958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:24.592397 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.592349    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt" podStartSLOduration=1.592335632 podStartE2EDuration="1.592335632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:37:24.588217974 +0000 UTC m=+3771.344626036" watchObservedRunningTime="2026-07-16 18:37:24.592335632 +0000 UTC m=+3771.348743692"
Jul 16 18:37:24.610733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.610695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k" podStartSLOduration=1.6106798329999998 podStartE2EDuration="1.610679833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:37:24.609464494 +0000 UTC m=+3771.365872555" watchObservedRunningTime="2026-07-16 18:37:24.610679833 +0000 UTC m=+3771.367087894"
Jul 16 18:37:24.876991 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:24.876912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:33.876731 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:33.876695    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:33.889605 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:33.889581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:34.619358 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:34.619329    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt"
Jul 16 18:37:44.694438 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:44.694406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:54.565336 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.565301    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:37:54.568376 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.568359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.576731 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.576710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 16 18:37:54.594777 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.594746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:37:54.596560 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.596528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:54.691190 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.691279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.691279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.691366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rrf\" (UniqueName: \"kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.691366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.691366 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.691362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.791850 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.791823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rrf\" (UniqueName: \"kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.791955 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.791856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.791955 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.791874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.791992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792033 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.792029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792121 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.792066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792260 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.792239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792383 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.792363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.792437 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.792394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.794122 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.794102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.794449 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.794433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.801861 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.801836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rrf\" (UniqueName: \"kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.878475 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.878415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:54.899870 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:54.899845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:55.009654 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.009629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:37:55.012084 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:37:55.012059    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb424524e_fa57_4ba0_991f_65d035c45887.slice/crio-e02b79c85c38bef499233d9c76c250dd391dc38a617dee821329323780e99c50 WatchSource:0}: Error finding container e02b79c85c38bef499233d9c76c250dd391dc38a617dee821329323780e99c50: Status 404 returned error can't find the container with id e02b79c85c38bef499233d9c76c250dd391dc38a617dee821329323780e99c50
Jul 16 18:37:55.571313 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.571287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k"
Jul 16 18:37:55.690905 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.690865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" event={"ID":"b424524e-fa57-4ba0-991f-65d035c45887","Type":"ContainerStarted","Data":"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"}
Jul 16 18:37:55.691068 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.690926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" event={"ID":"b424524e-fa57-4ba0-991f-65d035c45887","Type":"ContainerStarted","Data":"e02b79c85c38bef499233d9c76c250dd391dc38a617dee821329323780e99c50"}
Jul 16 18:37:55.719276 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.719219    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" podStartSLOduration=1.719198249 podStartE2EDuration="1.719198249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:37:55.718055925 +0000 UTC m=+3802.474463990" watchObservedRunningTime="2026-07-16 18:37:55.719198249 +0000 UTC m=+3802.475606311"
Jul 16 18:37:55.879079 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:55.879000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:37:57.249866 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:57.249835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:37:57.350467 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:37:57.350438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:38:04.879195 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:04.879157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:38:04.891154 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:04.891124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:38:05.743292 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:05.743268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:38:09.423270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:09.423239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:38:28.268562 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:28.268526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:38:28.275654 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:28.275627    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:38:28.327408 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:28.327379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:38:43.304940 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.304904    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" podUID="b424524e-fa57-4ba0-991f-65d035c45887" containerName="main" containerID="cri-o://9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b" gracePeriod=45
Jul 16 18:38:43.549153 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.549130    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:38:43.576959 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.576898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.576959 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.576953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.577113 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.576978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.577113 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.577113 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rrf\" (UniqueName: \"kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.577250 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs\") pod \"b424524e-fa57-4ba0-991f-65d035c45887\" (UID: \"b424524e-fa57-4ba0-991f-65d035c45887\") "
Jul 16 18:38:43.577308 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache" (OuterVolumeSpecName: "model-cache") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:38:43.577308 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:38:43.577308 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home" (OuterVolumeSpecName: "home") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:38:43.577476 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577392    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-model-cache\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.577476 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577407    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-home\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.577476 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.577422    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-tmp-dir\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.579051 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.579029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm" (OuterVolumeSpecName: "dshm") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:38:43.579225 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.579199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf" (OuterVolumeSpecName: "kube-api-access-s9rrf") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "kube-api-access-s9rrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:38:43.579266 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.579232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b424524e-fa57-4ba0-991f-65d035c45887" (UID: "b424524e-fa57-4ba0-991f-65d035c45887"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:38:43.678641 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.678610    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424524e-fa57-4ba0-991f-65d035c45887-tls-certs\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.678641 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.678635    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b424524e-fa57-4ba0-991f-65d035c45887-dshm\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.678641 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.678644    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rrf\" (UniqueName: \"kubernetes.io/projected/b424524e-fa57-4ba0-991f-65d035c45887-kube-api-access-s9rrf\") on node \"ip-10-0-142-99.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.899279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.899211    2577 generic.go:356] "Generic (PLEG): container finished" podID="b424524e-fa57-4ba0-991f-65d035c45887" containerID="9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b" exitCode=0
Jul 16 18:38:43.899279 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.899257    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"
Jul 16 18:38:43.899438 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.899297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" event={"ID":"b424524e-fa57-4ba0-991f-65d035c45887","Type":"ContainerDied","Data":"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"}
Jul 16 18:38:43.899438 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.899352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj" event={"ID":"b424524e-fa57-4ba0-991f-65d035c45887","Type":"ContainerDied","Data":"e02b79c85c38bef499233d9c76c250dd391dc38a617dee821329323780e99c50"}
Jul 16 18:38:43.899438 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.899374    2577 scope.go:117] "RemoveContainer" containerID="9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"
Jul 16 18:38:43.907770 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.907754    2577 scope.go:117] "RemoveContainer" containerID="9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"
Jul 16 18:38:43.908031 ip-10-0-142-99 kubenswrapper[2577]: E0716 18:38:43.908012    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b\": container with ID starting with 9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b not found: ID does not exist" containerID="9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"
Jul 16 18:38:43.908080 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.908043    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b"} err="failed to get container status \"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b\": rpc error: code = NotFound desc = could not find container \"9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b\": container with ID starting with 9e715625bd62429f1cd0b6790d243359119f4270d151949714fb9b6b2dc8449b not found: ID does not exist"
Jul 16 18:38:43.922220 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.922194    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:38:43.927614 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:43.927595    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc54jckj"]
Jul 16 18:38:44.305700 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:38:44.305666    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b424524e-fa57-4ba0-991f-65d035c45887" path="/var/lib/kubelet/pods/b424524e-fa57-4ba0-991f-65d035c45887/volumes"
Jul 16 18:41:29.303213 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:41:29.303165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:41:29.387303 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:41:29.387273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:44:36.817957 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:44:36.817918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:44:43.392553 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:44:43.392495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:44:43.456329 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:44:43.456300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:45:07.773126 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:45:07.773041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:45:39.783434 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:45:39.783400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:52:31.813529 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:52:31.813439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:54:28.906480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:54:28.906392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:56:23.615640 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:23.615611    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:23.629278 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:23.629253    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:26.998804 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:26.998765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:56:27.056241 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:27.056209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:56:42.661031 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:42.660999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:42.661546 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:42.661215    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:42.706657 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:42.706619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:42.799229 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:42.799167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:44.292971 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:44.292927    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:44.293619 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:44.293346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:44.336911 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:44.336872    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:44.414649 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:44.414612    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:45.883120 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:45.883090    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:45.883600 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:45.883533    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:45.930012 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:45.929985    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:46.007952 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:46.007934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:47.515044 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:47.515005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:47.515594 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:47.515396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:47.558300 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:47.558264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:47.635885 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:47.635854    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:49.132482 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:49.132453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:49.132955 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:49.132770    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:49.189682 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:49.189657    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:49.269930 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:49.269906    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:50.756044 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:50.756012    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:50.756684 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:50.756661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:50.802105 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:50.802088    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:50.877144 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:50.877121    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:52.374039 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:52.374009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:52.374522 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:52.374394    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:52.434692 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:52.434661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:52.515468 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:52.515436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:54.018752 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:54.018724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:54.019256 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:54.019202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:54.070985 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:54.070963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:54.150230 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:54.150208    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:55.941994 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:55.941903    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:55.942535 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:55.942283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:55.991410 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:55.991385    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:56.071801 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:56.071774    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:57.572078 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:57.572050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:57.572573 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:57.572393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:57.623708 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:57.623687    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:57.700084 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:57.700062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:56:59.179187 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:59.179157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:56:59.179634 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:59.179475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:56:59.221307 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:59.221281    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:56:59.296736 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:56:59.296713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:00.772674 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:00.772634    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:57:00.773247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:00.773210    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:57:00.834070 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:00.834046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:57:00.921243 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:00.921223    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:02.403630 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:02.403587    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:57:02.404237 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:02.404152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:57:02.449329 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:02.449304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:57:02.523789 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:02.523767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:04.018847 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:04.018806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:57:04.019270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:04.019217    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:57:04.064680 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:04.064655    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:57:04.143710 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:04.143668    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:05.644969 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:05.644936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:57:05.645426 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:05.645404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:57:05.692097 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:05.692075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:57:05.769738 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:05.769709    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:07.247156 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:07.247125    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/1.log"
Jul 16 18:57:07.247628 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:07.247577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-zbq44_811b18c8-7bd3-4780-8c4f-7662fbfd21e8/istio-proxy/0.log"
Jul 16 18:57:07.291402 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:07.291374    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-2jdrt_83cb5708-7fed-4211-9449-958f45154f9c/main/0.log"
Jul 16 18:57:07.372338 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:07.372307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-6ffwd5k_0683c982-6d4c-4bd1-aa07-1f6be8ea3e55/main/0.log"
Jul 16 18:57:14.160566 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:14.160531    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:14.192842 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:14.192817    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:15.452270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:15.452241    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:15.486645 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:15.486619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:16.765325 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:16.765291    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:16.796514 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:16.796477    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:18.010270 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:18.010239    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:18.039584 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:18.039557    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:19.258208 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:19.258175    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:19.290559 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:19.290532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:20.546539 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:20.546490    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:20.581293 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:20.581262    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:21.818937 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:21.818908    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:21.848249 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:21.848221    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:23.092069 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:23.092035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:23.123072 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:23.123038    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:24.349933 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:24.349904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:24.380843 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:24.380808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:25.602797 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:25.602758    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:25.635477 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:25.635447    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:26.869733 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:26.869702    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:26.901790 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:26.901752    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:29.585017 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:29.584967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:29.761416 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:29.761380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:29.862625 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:29.862544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:30.558821 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:30.558786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:30.659228 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:30.659178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:32.592549 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:32.587838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-785gt_4e6da779-aad7-4536-a3aa-30ae688b10a4/global-pull-secret-syncer/0.log"
Jul 16 18:57:32.718198 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:32.718173    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log"
Jul 16 18:57:32.898588 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:32.898488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-99.ec2.internal_8c50785a27e73d42664b01bc31a18dac/haproxy/0.log"
Jul 16 18:57:35.989552 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:35.989518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:36.898714 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:36.898684    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zcvdz_87352b8f-8830-4f09-aa00-b144c5a1b06e/authorino/0.log"
Jul 16 18:57:36.981534 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:36.981488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-7cst2_260439f5-dcdb-498e-b708-760078aa7e35/manager/0.log"
Jul 16 18:57:37.154822 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:37.154737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hmls_38cbd90d-23de-4ce3-b04c-f18a1bee0051/konnectivity-agent/0.log" limit=1024
Jul 16 18:57:39.058294 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.058255    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6587bc5797-tkxtf_3496c976-cb64-46fa-8987-3fefef17ac59/metrics-server/0.log"
Jul 16 18:57:39.200575 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.200488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnm77_8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f/node-exporter/0.log"
Jul 16 18:57:39.222310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.222280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnm77_8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f/kube-rbac-proxy/0.log"
Jul 16 18:57:39.244549 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.244533    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnm77_8d99c32a-04f2-4678-ad1f-dba4d9ec7f4f/init-textfile/0.log"
Jul 16 18:57:39.477902 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.477803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/prometheus/0.log"
Jul 16 18:57:39.495961 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.495944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/config-reloader/0.log"
Jul 16 18:57:39.520788 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.520761    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/thanos-sidecar/0.log"
Jul 16 18:57:39.545657 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.545640    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/kube-rbac-proxy-web/0.log"
Jul 16 18:57:39.571029 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.571011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/kube-rbac-proxy/0.log"
Jul 16 18:57:39.596784 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.596768    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/kube-rbac-proxy-thanos/0.log"
Jul 16 18:57:39.624527 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:39.624513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2a753b5-db7a-4438-9e1b-66e0730df992/init-config-reloader/0.log"
Jul 16 18:57:42.774931 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.774903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"]
Jul 16 18:57:42.775310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.775196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b424524e-fa57-4ba0-991f-65d035c45887" containerName="main"
Jul 16 18:57:42.775310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.775207    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b424524e-fa57-4ba0-991f-65d035c45887" containerName="main"
Jul 16 18:57:42.775310 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.775271    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b424524e-fa57-4ba0-991f-65d035c45887" containerName="main"
Jul 16 18:57:42.777973 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.777953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.780420 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.780401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt8tj\"/\"openshift-service-ca.crt\""
Jul 16 18:57:42.781233 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.781211    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kt8tj\"/\"default-dockercfg-px8tg\""
Jul 16 18:57:42.781328 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.781210    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt8tj\"/\"kube-root-ca.crt\""
Jul 16 18:57:42.793973 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.793952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"]
Jul 16 18:57:42.828331 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.828309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-podres\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.828437 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.828387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-proc\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.828437 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.828415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-lib-modules\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.828583 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.828449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5qd\" (UniqueName: \"kubernetes.io/projected/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-kube-api-access-7b5qd\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.828583 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.828490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-sys\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.928991 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.928971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-proc\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.928999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-lib-modules\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5qd\" (UniqueName: \"kubernetes.io/projected/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-kube-api-access-7b5qd\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-sys\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-proc\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-podres\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-lib-modules\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-podres\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.929205 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.929179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-sys\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:42.939247 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:42.939222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5qd\" (UniqueName: \"kubernetes.io/projected/8c1a1c2c-e761-4a4d-bec9-b48351026bd5-kube-api-access-7b5qd\") pod \"perf-node-gather-daemonset-5fvqp\" (UID: \"8c1a1c2c-e761-4a4d-bec9-b48351026bd5\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:43.089087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.089036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:43.215480 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.215454    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"]
Jul 16 18:57:43.217706 ip-10-0-142-99 kubenswrapper[2577]: W0716 18:57:43.217684    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c1a1c2c_e761_4a4d_bec9_b48351026bd5.slice/crio-f3545d9c4d9c9a3c933d78b924890c839635ede8d59a5fd25ed584256c0d7776 WatchSource:0}: Error finding container f3545d9c4d9c9a3c933d78b924890c839635ede8d59a5fd25ed584256c0d7776: Status 404 returned error can't find the container with id f3545d9c4d9c9a3c933d78b924890c839635ede8d59a5fd25ed584256c0d7776
Jul 16 18:57:43.398694 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.398617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp" event={"ID":"8c1a1c2c-e761-4a4d-bec9-b48351026bd5","Type":"ContainerStarted","Data":"97df5cb79628625fc7f8c6ee9f1af3f484fa5b4f0b9263a9870d191a6386ef6f"}
Jul 16 18:57:43.398694 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.398663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp" event={"ID":"8c1a1c2c-e761-4a4d-bec9-b48351026bd5","Type":"ContainerStarted","Data":"f3545d9c4d9c9a3c933d78b924890c839635ede8d59a5fd25ed584256c0d7776"}
Jul 16 18:57:43.398694 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.398692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:43.414688 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:43.414636    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp" podStartSLOduration=1.414622799 podStartE2EDuration="1.414622799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:57:43.413599671 +0000 UTC m=+4990.170007734" watchObservedRunningTime="2026-07-16 18:57:43.414622799 +0000 UTC m=+4990.171030861"
Jul 16 18:57:44.256095 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:44.256067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6rxh_c5885fde-e5b3-423c-9442-990136b5a521/dns/0.log"
Jul 16 18:57:44.277948 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:44.277925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6rxh_c5885fde-e5b3-423c-9442-990136b5a521/kube-rbac-proxy/0.log"
Jul 16 18:57:44.413988 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:44.413961    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nfsf7_658633e6-f491-4234-b88e-482a65cdab95/dns-node-resolver/0.log"
Jul 16 18:57:45.059420 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:45.059387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fq5kb_3b8f3865-fae0-4b1b-8a8f-92f8776798b6/node-ca/0.log"
Jul 16 18:57:47.044599 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.044568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dmmld_0a3dce21-336b-42fb-81b0-59104d77c969/serve-healthcheck-canary/0.log"
Jul 16 18:57:47.568976 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.568944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-psmpz_49b1cdcc-0bd7-491b-bac9-26b8e763a2dd/insights-operator/0.log"
Jul 16 18:57:47.571148 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.571122    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-psmpz_49b1cdcc-0bd7-491b-bac9-26b8e763a2dd/insights-operator/1.log"
Jul 16 18:57:47.798678 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.798644    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9zbv_eaa3c374-845e-4279-9205-9cf5bd8597fa/kube-rbac-proxy/0.log"
Jul 16 18:57:47.821972 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.821907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9zbv_eaa3c374-845e-4279-9205-9cf5bd8597fa/exporter/0.log"
Jul 16 18:57:47.845436 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:47.845420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9zbv_eaa3c374-845e-4279-9205-9cf5bd8597fa/extractor/0.log"
Jul 16 18:57:49.412045 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:49.412017    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-5fvqp"
Jul 16 18:57:50.559562 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:50.559523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c87b7c6dd-ln2vp_5f574320-981e-484e-ae71-89977f84d9f7/manager/0.log"
Jul 16 18:57:51.349523 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:51.349474    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-29xh8_6c44b3ca-4928-4287-b788-a069ed82cfec/server/0.log"
Jul 16 18:57:52.259815 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:52.259778    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-w7gqq_fe1e8698-c856-4200-8deb-08335f37b1ee/manager/0.log"
Jul 16 18:57:58.908572 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:58.908476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/kube-multus-additional-cni-plugins/0.log"
Jul 16 18:57:58.930428 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:58.930407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/egress-router-binary-copy/0.log"
Jul 16 18:57:58.951837 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:58.951817    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/cni-plugins/0.log"
Jul 16 18:57:58.973208 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:58.973190    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/bond-cni-plugin/0.log"
Jul 16 18:57:58.995743 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:58.995727    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/routeoverride-cni/0.log"
Jul 16 18:57:59.019087 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:59.019069    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/whereabouts-cni-bincopy/0.log"
Jul 16 18:57:59.040529 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:59.040498    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bskg_e7cbbeb8-9ff3-4e97-bc8d-5a7c784415ac/whereabouts-cni/0.log"
Jul 16 18:57:59.320139 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:59.320116    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zbvvt_ae688bb9-84d2-4be9-add5-340b787a7b4e/kube-multus/0.log"
Jul 16 18:57:59.445974 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:59.445955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tgz85_9e650ce2-5cf1-4786-8905-9c4775e5c64c/network-metrics-daemon/0.log"
Jul 16 18:57:59.464780 ip-10-0-142-99 kubenswrapper[2577]: I0716 18:57:59.464762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tgz85_9e650ce2-5cf1-4786-8905-9c4775e5c64c/kube-rbac-proxy/0.log"