Sep 17 18:29:06.443710 ip-10-0-132-9 systemd[1]: Starting Kubernetes Kubelet... Sep 17 18:29:09.767618 ip-10-0-132-9 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:09.767618 ip-10-0-132-9 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 18:29:09.767618 ip-10-0-132-9 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:09.767618 ip-10-0-132-9 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 18:29:09.767618 ip-10-0-132-9 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:09.768361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.767686 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 18:29:09.772051 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772029 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.772051 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772046 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.772051 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772050 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.772051 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772053 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.772051 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772057 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772060 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772063 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772067 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772070 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772073 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772076 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772079 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772082 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772085 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772087 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772090 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772093 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772095 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772098 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772101 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772103 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772106 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772109 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772114 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.772245 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772117 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772120 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772122 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772125 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772128 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772130 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772133 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772136 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772138 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772141 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772143 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772146 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772149 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772152 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772156 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772166 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772169 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772172 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772177 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772181 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.772746 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772185 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772188 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772190 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772193 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772195 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772198 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772200 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772203 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772206 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772209 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772211 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772214 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772216 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772219 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772222 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772224 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772227 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772229 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772232 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772235 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.773264 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772237 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772240 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772242 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772245 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772247 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772250 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772254 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772257 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772260 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772263 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772265 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772268 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772271 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772273 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772276 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772279 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772281 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772285 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772288 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772291 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.773774 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772293 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772296 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772298 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772301 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772303 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772306 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772309 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772311 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772314 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772319 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772323 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772326 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772329 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772333 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.772337 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773028 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773035 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773038 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773041 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.774246 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773044 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773047 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773049 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773052 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773054 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773057 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773059 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773062 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773064 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773069 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773073 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773076 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773079 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773082 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773085 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773088 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773090 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773092 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773095 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.774711 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773097 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773100 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773103 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773105 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773108 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773111 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773113 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773116 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773118 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773121 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773123 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773127 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773129 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773132 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773134 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773137 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773139 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773142 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773145 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773147 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.775174 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773150 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773153 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773155 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773158 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773160 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773163 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773167 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773174 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773177 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773180 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773182 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773187 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773189 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773192 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773195 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773197 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773200 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773202 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773206 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.775675 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773209 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773211 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773214 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773217 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773220 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773222 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773225 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773227 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773230 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773232 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773235 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773237 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773240 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773243 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773245 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773249 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773251 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773254 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773256 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773259 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.776136 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773261 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773267 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773270 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773275 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773280 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773283 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773286 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773289 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773292 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773294 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773297 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773300 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773302 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773305 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773307 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773310 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773312 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773698 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773704 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.776641 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773706 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773710 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773715 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773720 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773723 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773726 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773729 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773732 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773735 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773738 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773741 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773743 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773746 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773748 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773757 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773759 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773762 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773767 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773769 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.777092 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773772 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773774 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773778 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773781 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773783 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773786 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773789 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773791 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773794 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773797 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773800 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773802 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773805 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773807 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773810 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773812 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773815 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773817 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773820 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.777648 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773823 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773825 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773827 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773831 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773833 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773836 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773839 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773841 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773844 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773847 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773850 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773853 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773856 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773858 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773861 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773863 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773866 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773868 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773871 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773873 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.778110 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773876 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773879 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773881 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773884 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773886 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773889 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773891 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773894 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773896 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773899 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773901 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773903 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773906 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773908 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773911 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773913 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773916 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773918 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773921 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773923 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.778604 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773926 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773928 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773931 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773937 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773940 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773944 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773947 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773950 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773952 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773955 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773957 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773960 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773962 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773965 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773967 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773970 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773973 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773975 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.773978 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774050 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 18:29:09.779085 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774092 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774100 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774104 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774108 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774111 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774114 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774118 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774122 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774125 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774128 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774131 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774134 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774137 2559 flags.go:64] FLAG: --cgroup-root="" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774140 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774145 2559 flags.go:64] FLAG: --client-ca-file="" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774148 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774151 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774155 2559 flags.go:64] FLAG: --cluster-domain="" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774158 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774161 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774164 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774167 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774171 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 18:29:09.779568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774174 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774177 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774180 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774183 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774186 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774189 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774192 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774195 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774199 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774202 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774205 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774208 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774210 2559 flags.go:64] FLAG: --enable-server="true" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774213 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774217 2559 flags.go:64] FLAG: --event-burst="100" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774220 2559 flags.go:64] FLAG: --event-qps="50" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774223 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774226 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774229 2559 flags.go:64] FLAG: --eviction-hard="" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774232 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774235 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774238 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774241 2559 flags.go:64] FLAG: --eviction-soft="" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774245 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774248 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 18:29:09.780110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774252 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774255 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774258 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774261 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774264 2559 flags.go:64] FLAG: --feature-gates="" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774267 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774270 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774273 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774276 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774279 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774282 2559 flags.go:64] FLAG: --help="false" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774284 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-132-9.ec2.internal" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774288 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774291 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774293 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774296 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774300 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774302 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774305 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774308 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774310 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774313 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774316 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774319 2559 flags.go:64] FLAG: --kube-reserved="" Sep 17 18:29:09.780755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774322 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774324 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774327 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774330 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774333 2559 flags.go:64] FLAG: --lock-file="" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774335 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774340 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774344 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774349 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774353 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774356 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774358 2559 flags.go:64] FLAG: --logging-format="text" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774362 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774365 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774367 2559 flags.go:64] FLAG: --manifest-url="" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774370 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774374 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774377 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774381 2559 flags.go:64] FLAG: --max-pods="110" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774384 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774387 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774389 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774393 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774396 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774399 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 18:29:09.781408 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774401 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774409 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774411 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774414 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774417 2559 flags.go:64] FLAG: --pod-cidr="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774420 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774425 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774427 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774430 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774433 2559 flags.go:64] FLAG: --port="10250" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774436 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774439 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06180b82520961029" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774442 2559 flags.go:64] FLAG: --qos-reserved="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774459 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774462 2559 flags.go:64] FLAG: --register-node="true" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774467 2559 flags.go:64] FLAG: --register-with-taints="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774470 2559 flags.go:64] FLAG: --registry-burst="10" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774474 2559 flags.go:64] FLAG: --registry-qps="5" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774477 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774480 2559 flags.go:64] FLAG: --reserved-memory="" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774483 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774486 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774489 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774492 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774495 2559 flags.go:64] FLAG: --runonce="false" Sep 17 18:29:09.782027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774498 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774501 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774504 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774507 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774510 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774513 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774516 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774519 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774522 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774525 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774527 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774530 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774533 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774536 2559 flags.go:64] FLAG: --system-cgroups="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774539 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774544 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774547 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774550 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774553 2559 flags.go:64] FLAG: --tls-min-version="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774556 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774560 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774563 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774567 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774570 2559 flags.go:64] FLAG: --v="2" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774573 2559 flags.go:64] FLAG: --version="false" Sep 17 18:29:09.782657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774584 2559 flags.go:64] FLAG: --vmodule="" Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774587 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.774591 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774721 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774725 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774728 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774731 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774733 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774736 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774739 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774742 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774744 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774747 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774750 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774753 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774755 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774758 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774761 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774763 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774766 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.783255 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774768 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774771 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774774 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774777 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774779 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774782 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774784 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774788 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774791 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774795 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774806 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774809 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774812 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774815 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774818 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774821 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774823 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774826 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774829 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774832 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.783791 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774834 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774837 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774839 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774844 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774847 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774851 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774854 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774857 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774860 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774862 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774865 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774868 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774871 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774874 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774877 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774880 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774882 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774885 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774888 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.784285 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774891 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774894 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774899 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774903 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774906 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774909 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774912 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774915 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774918 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774921 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774924 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774927 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774930 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774933 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774935 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774938 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774940 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774943 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774945 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.784757 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774948 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774951 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774953 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774957 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774960 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774963 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774965 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774968 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774971 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774973 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774976 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774978 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774981 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774985 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774988 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774992 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774994 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.774997 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775001 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.785205 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775003 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775006 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775009 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775012 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.775014 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.775544 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.782571 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.782591 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.782609 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.782617 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782668 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782674 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782677 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782680 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.785718 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782684 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782688 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782691 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782694 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782697 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782701 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782704 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782706 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782709 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782712 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782714 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782717 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782720 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782722 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782725 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782727 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782730 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782733 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782736 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782738 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.786066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782741 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782744 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782748 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782750 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782753 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782757 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782760 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782763 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782765 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782768 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782770 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782773 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782775 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782778 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782781 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782784 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782787 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782789 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782792 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.786598 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782795 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782797 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782800 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782803 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782805 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782807 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782810 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782813 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782815 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782819 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782822 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782825 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782828 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782830 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782833 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782836 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782838 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782841 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782843 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.787052 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782847 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782849 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782852 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782855 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782857 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782860 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782862 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782865 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782867 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782870 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782872 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782875 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782877 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782880 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782882 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782885 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782887 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782890 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782892 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782895 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.787526 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782897 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782900 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782903 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782906 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782909 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782911 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782914 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782916 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782919 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782921 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782924 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782927 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782930 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782933 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782935 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782938 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.788010 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.782940 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.782945 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783052 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783057 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783060 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783062 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783065 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783067 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783070 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783072 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783075 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783078 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783080 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783083 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:09.788390 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783087 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783091 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783094 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783096 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783099 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783103 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783106 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783109 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783112 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783115 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783117 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783120 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783123 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783126 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783128 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783131 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783134 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783137 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783139 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:09.788772 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783142 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783144 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783147 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783149 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783152 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783154 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783157 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783159 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783162 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783165 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783168 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783171 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783174 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783177 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783180 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783183 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783186 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783188 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783191 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783194 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:09.789223 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783197 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783199 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783202 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783204 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783207 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783209 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783212 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783214 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783217 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783219 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783222 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783225 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783227 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783230 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783232 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783235 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783237 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783240 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783242 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783245 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:09.789719 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783247 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783250 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783253 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783256 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783259 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783262 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783267 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783270 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783273 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783275 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783278 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783281 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783283 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783286 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783288 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783291 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783293 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783296 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783299 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:09.790229 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783301 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783304 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783307 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783309 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783312 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783315 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783317 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783320 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:09.783323 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.783327 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.783413 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.785365 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.786400 2559 server.go:1019] "Starting client certificate rotation" Sep 17 18:29:09.790902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.786499 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:29:09.791559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.786529 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:29:09.791559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.788699 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:29:09.791559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.789931 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:29:09.800751 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.800731 2559 log.go:25] "Validated CRI v1 runtime API" Sep 17 18:29:09.802283 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.802270 2559 log.go:25] "Validated CRI v1 image API" Sep 17 18:29:09.803129 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.803109 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 18:29:09.806948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.806931 2559 fs.go:135] Filesystem UUIDs: map[659992aa-68ec-46b0-ac8f-e81356b9f242:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f9f6d0f9-22c5-43cd-9c3b-e4afe155f2e0:/dev/nvme0n1p3] Sep 17 18:29:09.807022 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.806947 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 18:29:09.812365 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.812242 2559 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:09.810489777 +0000 UTC m=+1.799106017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100886 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb7d5878205212374eaf03f8d5a07 SystemUUID:ec2cb7d5-8782-0521-2374-eaf03f8d5a07 BootID:601ec1f1-a255-4cc1-a1ba-06f835833cab Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:22:1e:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:22:1e:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:1a:4a:82:f8:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 18:29:09.812365 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.812354 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 18:29:09.812534 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.812466 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 18:29:09.814257 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.814235 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 18:29:09.814413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.814260 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 18:29:09.815492 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.815481 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 18:29:09.815525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.815495 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 18:29:09.815525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.815506 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:29:09.815525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.815516 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:29:09.816965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.816954 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 18:29:09.820253 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.820236 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:29:09.820881 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.820870 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:29:09.821147 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821136 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 18:29:09.821178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821172 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 17 18:29:09.821208 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821186 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 18:29:09.821208 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821201 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 18:29:09.821208 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821209 2559 kubelet.go:398] "Adding apiserver pod source" Sep 17 18:29:09.821298 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821218 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 18:29:09.821571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821550 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:29:09.821625 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.821578 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:29:09.829858 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.829840 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 18:29:09.830475 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.830437 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 18:29:09.830601 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.830566 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 18:29:09.834897 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.834878 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 18:29:09.834975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.834911 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 18:29:09.836309 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836298 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836314 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836320 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836325 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836333 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836339 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 18:29:09.836352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836351 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836357 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836363 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836368 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836380 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836385 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836407 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 18:29:09.836525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.836412 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 18:29:09.839124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.839109 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 18:29:09.839928 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.839916 2559 server.go:1268] "Started kubelet" Sep 17 18:29:09.840047 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.839963 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 18:29:09.840133 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.840022 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 18:29:09.840133 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.840083 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 18:29:09.840949 ip-10-0-132-9 systemd[1]: Started Kubernetes Kubelet. Sep 17 18:29:09.841877 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.841424 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 17 18:29:09.842030 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.841907 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 18:29:09.845920 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.845898 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 18:29:09.848999 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.848986 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 18:29:09.849052 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849002 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 18:29:09.849107 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.849051 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:09.849523 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849504 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 18:29:09.849523 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849523 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 18:29:09.849733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849720 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 17 18:29:09.849733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849730 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 17 18:29:09.849851 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.849752 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 18:29:09.852734 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852710 2559 factory.go:55] Registering systemd factory Sep 17 18:29:09.852734 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852722 2559 factory.go:223] Registration of the systemd container factory successfully Sep 17 18:29:09.853042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852918 2559 factory.go:153] Registering CRI-O factory Sep 17 18:29:09.853042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852928 2559 factory.go:223] Registration of the crio container factory successfully Sep 17 18:29:09.853042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852973 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 18:29:09.853042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.852992 2559 factory.go:103] Registering Raw factory Sep 17 18:29:09.853042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.853004 2559 manager.go:1196] Started watching for new ooms in manager Sep 17 18:29:09.853491 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.853473 2559 manager.go:349] Starting recovery of all containers Sep 17 18:29:09.857459 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.857420 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 18:29:09.857554 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.857429 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 18:29:09.893430 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.857521 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-9.ec2.internal.18d62f27c877f8b0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-9.ec2.internal,UID:ip-10-0-132-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-9.ec2.internal,},FirstTimestamp:2026-09-17 18:29:09.839886512 +0000 UTC m=+1.828502753,LastTimestamp:2026-09-17 18:29:09.839886512 +0000 UTC m=+1.828502753,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-9.ec2.internal,}" Sep 17 18:29:09.893552 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.893462 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-czkt7" Sep 17 18:29:09.893552 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.893500 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-czkt7" Sep 17 18:29:09.896911 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.896891 2559 manager.go:354] Recovery completed Sep 17 18:29:09.901541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.901528 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:09.903296 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903279 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:09.903372 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903306 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:09.903372 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903317 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:09.903851 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903835 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 18:29:09.903851 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903848 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 18:29:09.903851 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903849 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:09.903992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.903861 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:29:09.904538 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.904521 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:09.904632 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.904551 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:09.904632 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.904565 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:09.905751 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.905656 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-9.ec2.internal.18d62f27cc3f815d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-9.ec2.internal,UID:ip-10-0-132-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-9.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-9.ec2.internal,},FirstTimestamp:2026-09-17 18:29:09.903294813 +0000 UTC m=+1.891911053,LastTimestamp:2026-09-17 18:29:09.903294813 +0000 UTC m=+1.891911053,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-9.ec2.internal,}" Sep 17 18:29:09.905835 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.905799 2559 policy_none.go:49] "None policy: Start" Sep 17 18:29:09.905835 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.905814 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 18:29:09.905835 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.905826 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 18:29:09.907790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.907770 2559 policy_none.go:47] "Start" Sep 17 18:29:09.946988 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.946972 2559 manager.go:337] "Starting Device Plugin manager" Sep 17 18:29:09.947116 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.947011 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 18:29:09.947116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947026 2559 server.go:87] "Starting device plugin registration server" Sep 17 18:29:09.947281 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947268 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 18:29:09.947344 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947291 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 18:29:09.947393 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947348 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 18:29:09.947459 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947428 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 18:29:09.947459 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.947438 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 18:29:09.947857 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.947837 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 18:29:09.947956 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.947881 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:09.985480 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.985462 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 18:29:09.986612 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.986589 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 18:29:09.986699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.986642 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 18:29:09.986699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.986664 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 18:29:09.986800 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:09.986706 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 18:29:09.988764 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:09.988740 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:10.048363 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.048322 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:10.049138 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.049124 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:10.049194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.049149 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:10.049194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.049158 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:10.049194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.049181 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.058442 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.058429 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.058512 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.058460 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-9.ec2.internal\": node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.083577 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.083558 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.087623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.087609 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal"] Sep 17 18:29:10.087673 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.087659 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:10.088375 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.088362 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:10.088441 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.088385 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:10.088441 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.088399 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:10.090553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.090541 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:10.090673 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.090660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.090726 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.090689 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:10.091169 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091154 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:10.091235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091173 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:10.091235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091182 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:10.091235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091197 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:10.091350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091197 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:10.091350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.091269 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:10.093620 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.093606 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.093706 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.093633 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:10.094260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.094229 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:10.094332 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.094265 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:10.094332 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.094275 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:10.115039 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.115022 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-9.ec2.internal\" not found" node="ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.119202 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.119189 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-9.ec2.internal\" not found" node="ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.151251 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.151233 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb7ea012701865f5588b4ef5f36a7e27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"cb7ea012701865f5588b4ef5f36a7e27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.151320 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.151254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.151320 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.151270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.184072 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.184053 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.251501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251480 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.251568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.251568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb7ea012701865f5588b4ef5f36a7e27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"cb7ea012701865f5588b4ef5f36a7e27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.251568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb7ea012701865f5588b4ef5f36a7e27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"cb7ea012701865f5588b4ef5f36a7e27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.251658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.251658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.251572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc2e9e1a999339d953e286eb83e762e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"4bc2e9e1a999339d953e286eb83e762e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.284608 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.284580 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.385287 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.385246 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.418398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.418383 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.423399 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.423385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.486123 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.486095 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.586616 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.586594 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.687104 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.687058 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.773386 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.773370 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:10.785938 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.785917 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 18:29:10.786050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.786034 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 18:29:10.786107 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.786067 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 18:29:10.788048 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:10.788034 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 17 18:29:10.880705 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.880685 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:10.899660 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.899632 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:09 +0000 UTC" deadline="2028-06-14 08:18:25.978489246 +0000 UTC" Sep 17 18:29:10.899660 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.899657 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15253h49m15.078834304s" Sep 17 18:29:10.949972 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.949922 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.964197 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.964177 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:29:10.966886 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.965556 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 17 18:29:10.974988 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:10.974968 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:29:11.011680 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:11.011651 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7ea012701865f5588b4ef5f36a7e27.slice/crio-59cb5a66a551412cbb1a7da3596fd23e576d45ea3a58db35b88d77d0573c1fb7 WatchSource:0}: Error finding container 59cb5a66a551412cbb1a7da3596fd23e576d45ea3a58db35b88d77d0573c1fb7: Status 404 returned error can't find the container with id 59cb5a66a551412cbb1a7da3596fd23e576d45ea3a58db35b88d77d0573c1fb7 Sep 17 18:29:11.012106 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:11.012086 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc2e9e1a999339d953e286eb83e762e.slice/crio-d3eed288895e01426aee0f58c8932d15543f926b936646347fbb837d34e7a976 WatchSource:0}: Error finding container d3eed288895e01426aee0f58c8932d15543f926b936646347fbb837d34e7a976: Status 404 returned error can't find the container with id d3eed288895e01426aee0f58c8932d15543f926b936646347fbb837d34e7a976 Sep 17 18:29:11.016316 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.016296 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:29:11.274572 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.274523 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:11.823249 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.823183 2559 apiserver.go:52] "Watching apiserver" Sep 17 18:29:11.829901 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.829869 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:11.831094 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.831074 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 18:29:11.831477 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.831431 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp","openshift-image-registry/node-ca-29dvp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal","openshift-multus/network-metrics-daemon-jq4tl","openshift-network-diagnostics/network-check-target-lx76w","openshift-network-operator/iptables-alerter-f2fg6","kube-system/global-pull-secret-syncer-lx2z8","openshift-cluster-node-tuning-operator/tuned-gd8cq","openshift-dns/node-resolver-j2lmq","openshift-multus/multus-additional-cni-plugins-g9dfm","openshift-multus/multus-g72sf","openshift-ovn-kubernetes/ovnkube-node-gbgqb","kube-system/konnectivity-agent-csc94"] Sep 17 18:29:11.834129 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.834112 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.836246 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.836207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.837075 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.837053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 18:29:11.837225 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.837206 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.837537 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.837519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdhmr\"" Sep 17 18:29:11.837655 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.837523 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.838267 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.838246 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.838717 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.838520 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.838717 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.838569 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 18:29:11.838717 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.838594 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:11.838939 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.838900 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dcp86\"" Sep 17 18:29:11.838939 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.838920 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.840961 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.840819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:11.840961 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.840900 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:11.842908 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.842846 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.845226 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.845075 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.845226 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.845163 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:11.845800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.845624 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 18:29:11.845800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.845634 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.845800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.845641 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-85fwd\"" Sep 17 18:29:11.845800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.845644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.847417 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.847393 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.849714 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.849695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.849807 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.849714 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.850603 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.850581 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 18:29:11.853114 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.851822 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.853114 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.852108 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gm2qz\"" Sep 17 18:29:11.853114 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.852852 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.854502 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.853897 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-slc89\"" Sep 17 18:29:11.854502 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.854326 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.854906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.854885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.857604 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857520 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.857604 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857561 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 18:29:11.858236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857798 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 18:29:11.858236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857834 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.858236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857916 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.858236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.857952 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 18:29:11.858236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.858174 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p2729\"" Sep 17 18:29:11.860182 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.860103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.860828 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.860663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w2n8w\"" Sep 17 18:29:11.860960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.860944 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 18:29:11.861986 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.861925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-systemd\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.861986 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.861959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-tmp\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.861986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-kubelet-config\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862009 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-conf\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-cnibin\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.862122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-socket-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-sys-fs\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscnv\" (UniqueName: \"kubernetes.io/projected/5a586aaa-a2e4-462e-8c3d-617310c54a19-kube-api-access-nscnv\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-kubernetes\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-sys\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-host\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn78m\" (UniqueName: \"kubernetes.io/projected/4583ef38-ee5c-4056-9acd-00782d3adf4b-kube-api-access-fn78m\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-device-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862361 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjh4l\" (UniqueName: \"kubernetes.io/projected/af33fe1c-3c60-4691-b297-97e78954f284-kube-api-access-hjh4l\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysconfig\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4n9\" (UniqueName: \"kubernetes.io/projected/1c2db46f-3b5d-43db-a562-7eedafad93ce-kube-api-access-9n4n9\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5011467-2009-4336-ac87-cd9bf72b1881-serviceca\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862497 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-var-lib-kubelet\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862514 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 18:29:11.862554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-dbus\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4583ef38-ee5c-4056-9acd-00782d3adf4b-hosts-file\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-registration-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdgg\" (UniqueName: \"kubernetes.io/projected/e08ab8d9-eb73-4356-b19c-3250af393478-kube-api-access-8gdgg\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-system-cni-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-run\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-lib-modules\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-tuned\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4583ef38-ee5c-4056-9acd-00782d3adf4b-tmp-dir\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5011467-2009-4336-ac87-cd9bf72b1881-host\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a586aaa-a2e4-462e-8c3d-617310c54a19-iptables-alerter-script\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-modprobe-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxv85\" (UniqueName: \"kubernetes.io/projected/aa4163b5-6ed7-4af9-a018-c265369dda47-kube-api-access-dxv85\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.863361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.862985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-etc-selinux\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqd52\" (UniqueName: \"kubernetes.io/projected/f5011467-2009-4336-ac87-cd9bf72b1881-kube-api-access-mqd52\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a586aaa-a2e4-462e-8c3d-617310c54a19-host-slash\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863752 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863749 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f8xjr\"" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.863932 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 18:29:11.864078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.864057 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 18:29:11.864462 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.864118 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 18:29:11.864462 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.864331 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:29:11.864462 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.864377 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:11.866880 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.866858 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v565h\"" Sep 17 18:29:11.867033 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.866858 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 18:29:11.867033 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.866922 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 18:29:11.883844 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.883824 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-grjrk" Sep 17 18:29:11.890142 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.890126 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-grjrk" Sep 17 18:29:11.951474 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.951255 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 18:29:11.963742 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-socket-dir-parent\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.963841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-bin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.963841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-registration-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.963841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdgg\" (UniqueName: \"kubernetes.io/projected/e08ab8d9-eb73-4356-b19c-3250af393478-kube-api-access-8gdgg\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.963841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-system-cni-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963868 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-registration-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xtp\" (UniqueName: \"kubernetes.io/projected/dd4073a8-521e-483a-abb9-0506b859d73c-kube-api-access-n4xtp\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-lib-modules\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-system-cni-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.963993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-tuned\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964019 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-lib-modules\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-conf-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fe05e5e-1aff-454a-9409-585a30fc2aae-konnectivity-ca\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5011467-2009-4336-ac87-cd9bf72b1881-host\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5011467-2009-4336-ac87-cd9bf72b1881-host\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-systemd-units\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-netd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964283 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 18:29:11.964367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-etc-selinux\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964381 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-etc-selinux\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqd52\" (UniqueName: \"kubernetes.io/projected/f5011467-2009-4336-ac87-cd9bf72b1881-kube-api-access-mqd52\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-systemd\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-tmp\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-systemd\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-kubelet\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-etc-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-sys-fs\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscnv\" (UniqueName: \"kubernetes.io/projected/5a586aaa-a2e4-462e-8c3d-617310c54a19-kube-api-access-nscnv\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-kubernetes\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.964704 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-sys\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-host\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.964918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964790 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-netns\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-netns\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-device-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysconfig\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-system-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.964908 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.464888664 +0000 UTC m=+4.453504905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-sys-fs\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-sys\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-cni-binary-copy\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysconfig\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964986 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-kubernetes\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.964968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-device-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-host\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-multus-daemon-config\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965095 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-script-lib\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.965641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5011467-2009-4336-ac87-cd9bf72b1881-serviceca\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-dbus\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4583ef38-ee5c-4056-9acd-00782d3adf4b-hosts-file\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-etc-kubernetes\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-kubelet\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965269 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-hostroot\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-ovn\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4583ef38-ee5c-4056-9acd-00782d3adf4b-hosts-file\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-run\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4583ef38-ee5c-4056-9acd-00782d3adf4b-tmp-dir\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-k8s-cni-cncf-io\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-dbus\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965420 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-systemd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.966414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965481 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllv2\" (UniqueName: \"kubernetes.io/projected/d409999b-ee91-4485-868b-4162e1b49553-kube-api-access-fllv2\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a586aaa-a2e4-462e-8c3d-617310c54a19-iptables-alerter-script\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-run\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-modprobe-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5011467-2009-4336-ac87-cd9bf72b1881-serviceca\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxv85\" (UniqueName: \"kubernetes.io/projected/aa4163b5-6ed7-4af9-a018-c265369dda47-kube-api-access-dxv85\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-multus\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-slash\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-modprobe-d\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4583ef38-ee5c-4056-9acd-00782d3adf4b-tmp-dir\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-log-socket\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a586aaa-a2e4-462e-8c3d-617310c54a19-host-slash\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-kubelet-config\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.965977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-cnibin\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-node-log\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966070 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-bin\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a586aaa-a2e4-462e-8c3d-617310c54a19-host-slash\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea05b683-b14d-4b78-b315-f3daaed894e0-kubelet-config\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d409999b-ee91-4485-868b-4162e1b49553-ovn-node-metrics-cert\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-conf\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-cnibin\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c2db46f-3b5d-43db-a562-7eedafad93ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-socket-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn78m\" (UniqueName: \"kubernetes.io/projected/4583ef38-ee5c-4056-9acd-00782d3adf4b-kube-api-access-fn78m\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-sysctl-conf\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4n9\" (UniqueName: \"kubernetes.io/projected/1c2db46f-3b5d-43db-a562-7eedafad93ce-kube-api-access-9n4n9\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966351 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-cnibin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-multus-certs\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966465 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af33fe1c-3c60-4691-b297-97e78954f284-socket-dir\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-var-lib-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-config\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.967889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjh4l\" (UniqueName: \"kubernetes.io/projected/af33fe1c-3c60-4691-b297-97e78954f284-kube-api-access-hjh4l\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a586aaa-a2e4-462e-8c3d-617310c54a19-iptables-alerter-script\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966635 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-env-overrides\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fe05e5e-1aff-454a-9409-585a30fc2aae-agent-certs\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.966716 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-var-lib-kubelet\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa4163b5-6ed7-4af9-a018-c265369dda47-var-lib-kubelet\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.966779 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.466763681 +0000 UTC m=+4.455379926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.966797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c2db46f-3b5d-43db-a562-7eedafad93ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.967679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-etc-tuned\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.968499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.967730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4163b5-6ed7-4af9-a018-c265369dda47-tmp\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.974069 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.974051 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:11.974218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.974077 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:11.974218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.974091 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:11.974218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:11.974149 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.474134466 +0000 UTC m=+4.462750695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:11.975175 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.975105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdgg\" (UniqueName: \"kubernetes.io/projected/e08ab8d9-eb73-4356-b19c-3250af393478-kube-api-access-8gdgg\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:11.976028 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.976002 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4n9\" (UniqueName: \"kubernetes.io/projected/1c2db46f-3b5d-43db-a562-7eedafad93ce-kube-api-access-9n4n9\") pod \"multus-additional-cni-plugins-g9dfm\" (UID: \"1c2db46f-3b5d-43db-a562-7eedafad93ce\") " pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:11.977524 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.977500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn78m\" (UniqueName: \"kubernetes.io/projected/4583ef38-ee5c-4056-9acd-00782d3adf4b-kube-api-access-fn78m\") pod \"node-resolver-j2lmq\" (UID: \"4583ef38-ee5c-4056-9acd-00782d3adf4b\") " pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:11.977907 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.977887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxv85\" (UniqueName: \"kubernetes.io/projected/aa4163b5-6ed7-4af9-a018-c265369dda47-kube-api-access-dxv85\") pod \"tuned-gd8cq\" (UID: \"aa4163b5-6ed7-4af9-a018-c265369dda47\") " pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:11.978183 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.978087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscnv\" (UniqueName: \"kubernetes.io/projected/5a586aaa-a2e4-462e-8c3d-617310c54a19-kube-api-access-nscnv\") pod \"iptables-alerter-f2fg6\" (UID: \"5a586aaa-a2e4-462e-8c3d-617310c54a19\") " pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:11.978183 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.978129 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjh4l\" (UniqueName: \"kubernetes.io/projected/af33fe1c-3c60-4691-b297-97e78954f284-kube-api-access-hjh4l\") pod \"aws-ebs-csi-driver-node-74lrp\" (UID: \"af33fe1c-3c60-4691-b297-97e78954f284\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:11.979519 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.979501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqd52\" (UniqueName: \"kubernetes.io/projected/f5011467-2009-4336-ac87-cd9bf72b1881-kube-api-access-mqd52\") pod \"node-ca-29dvp\" (UID: \"f5011467-2009-4336-ac87-cd9bf72b1881\") " pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:11.991239 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.991188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" event={"ID":"cb7ea012701865f5588b4ef5f36a7e27","Type":"ContainerStarted","Data":"59cb5a66a551412cbb1a7da3596fd23e576d45ea3a58db35b88d77d0573c1fb7"} Sep 17 18:29:11.992313 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:11.992287 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"4bc2e9e1a999339d953e286eb83e762e","Type":"ContainerStarted","Data":"d3eed288895e01426aee0f58c8932d15543f926b936646347fbb837d34e7a976"} Sep 17 18:29:12.067474 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-system-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-cni-binary-copy\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-multus-daemon-config\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-script-lib\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-etc-kubernetes\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-system-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-kubelet\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-hostroot\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-etc-kubernetes\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-ovn\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-ovn\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-hostroot\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-k8s-cni-cncf-io\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-k8s-cni-cncf-io\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-kubelet\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-systemd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllv2\" (UniqueName: \"kubernetes.io/projected/d409999b-ee91-4485-868b-4162e1b49553-kube-api-access-fllv2\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-run-systemd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-multus\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-slash\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-log-socket\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-multus\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-node-log\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.067959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-bin\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067919 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-slash\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d409999b-ee91-4485-868b-4162e1b49553-ovn-node-metrics-cert\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-node-log\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-cnibin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.067990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-multus-certs\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-log-socket\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-var-lib-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-config\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-bin\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-cni-binary-copy\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd4073a8-521e-483a-abb9-0506b859d73c-multus-daemon-config\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-env-overrides\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fe05e5e-1aff-454a-9409-585a30fc2aae-agent-certs\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-multus-certs\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-socket-dir-parent\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-script-lib\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-bin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.068650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-cnibin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-var-lib-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-cni-bin\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-socket-dir-parent\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xtp\" (UniqueName: \"kubernetes.io/projected/dd4073a8-521e-483a-abb9-0506b859d73c-kube-api-access-n4xtp\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-conf-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fe05e5e-1aff-454a-9409-585a30fc2aae-konnectivity-ca\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-conf-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-multus-cni-dir\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-systemd-units\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-netd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-cni-netd\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-kubelet\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-ovnkube-config\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068535 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-etc-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-systemd-units\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-etc-openvswitch\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-var-lib-kubelet\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-netns\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-ovn-kubernetes\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4073a8-521e-483a-abb9-0506b859d73c-host-run-netns\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-netns\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d409999b-ee91-4485-868b-4162e1b49553-host-run-netns\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d409999b-ee91-4485-868b-4162e1b49553-env-overrides\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.069959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.068975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fe05e5e-1aff-454a-9409-585a30fc2aae-konnectivity-ca\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:12.070360 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.070274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d409999b-ee91-4485-868b-4162e1b49553-ovn-node-metrics-cert\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.070413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.070394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fe05e5e-1aff-454a-9409-585a30fc2aae-agent-certs\") pod \"konnectivity-agent-csc94\" (UID: \"2fe05e5e-1aff-454a-9409-585a30fc2aae\") " pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:12.077717 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.077669 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xtp\" (UniqueName: \"kubernetes.io/projected/dd4073a8-521e-483a-abb9-0506b859d73c-kube-api-access-n4xtp\") pod \"multus-g72sf\" (UID: \"dd4073a8-521e-483a-abb9-0506b859d73c\") " pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.077796 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.077779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllv2\" (UniqueName: \"kubernetes.io/projected/d409999b-ee91-4485-868b-4162e1b49553-kube-api-access-fllv2\") pod \"ovnkube-node-gbgqb\" (UID: \"d409999b-ee91-4485-868b-4162e1b49553\") " pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.146619 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.146600 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" Sep 17 18:29:12.161578 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.161558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-29dvp" Sep 17 18:29:12.172163 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.172139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f2fg6" Sep 17 18:29:12.177918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.177903 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" Sep 17 18:29:12.184923 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.184904 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j2lmq" Sep 17 18:29:12.194459 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.194437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" Sep 17 18:29:12.201207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.201194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g72sf" Sep 17 18:29:12.207964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.207946 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:12.214586 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.214571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:12.470260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.470118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:12.470260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.470198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:12.470442 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.470304 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:12.470442 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.470362 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:13.470349059 +0000 UTC m=+5.458965290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:12.470442 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.470304 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:12.470442 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.470436 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:13.470424014 +0000 UTC m=+5.459040256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:12.570809 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.570782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:12.570964 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.570886 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:12.570964 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.570899 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:12.570964 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.570907 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:12.570964 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.570955 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:13.570939375 +0000 UTC m=+5.559555620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:12.813638 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.813199 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4163b5_6ed7_4af9_a018_c265369dda47.slice/crio-6cdb517a33f0476aec22e4c6ca887aa2dfac7076b1a62862a79090619ce68b85 WatchSource:0}: Error finding container 6cdb517a33f0476aec22e4c6ca887aa2dfac7076b1a62862a79090619ce68b85: Status 404 returned error can't find the container with id 6cdb517a33f0476aec22e4c6ca887aa2dfac7076b1a62862a79090619ce68b85 Sep 17 18:29:12.814832 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.814793 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4583ef38_ee5c_4056_9acd_00782d3adf4b.slice/crio-08a957e2bd75dac90fd89986414462290ebb536f1fa736e123e215e586040b60 WatchSource:0}: Error finding container 08a957e2bd75dac90fd89986414462290ebb536f1fa736e123e215e586040b60: Status 404 returned error can't find the container with id 08a957e2bd75dac90fd89986414462290ebb536f1fa736e123e215e586040b60 Sep 17 18:29:12.819764 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.819689 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf33fe1c_3c60_4691_b297_97e78954f284.slice/crio-6e97af3d1597285197c49da9c5e8401b4d936b1a75a39d34ef96f9242a78ee49 WatchSource:0}: Error finding container 6e97af3d1597285197c49da9c5e8401b4d936b1a75a39d34ef96f9242a78ee49: Status 404 returned error can't find the container with id 6e97af3d1597285197c49da9c5e8401b4d936b1a75a39d34ef96f9242a78ee49 Sep 17 18:29:12.819983 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.819967 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe05e5e_1aff_454a_9409_585a30fc2aae.slice/crio-81da1ac21454f15d53a2da6b809aa286e9a3c412dfe0311388d9be457af96f74 WatchSource:0}: Error finding container 81da1ac21454f15d53a2da6b809aa286e9a3c412dfe0311388d9be457af96f74: Status 404 returned error can't find the container with id 81da1ac21454f15d53a2da6b809aa286e9a3c412dfe0311388d9be457af96f74 Sep 17 18:29:12.822917 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.822893 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5011467_2009_4336_ac87_cd9bf72b1881.slice/crio-1af3f3e012f7eee53897d1fb5a5d3ed66deb5a832a62e78c7d5201c73f7ca180 WatchSource:0}: Error finding container 1af3f3e012f7eee53897d1fb5a5d3ed66deb5a832a62e78c7d5201c73f7ca180: Status 404 returned error can't find the container with id 1af3f3e012f7eee53897d1fb5a5d3ed66deb5a832a62e78c7d5201c73f7ca180 Sep 17 18:29:12.823840 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.823816 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2db46f_3b5d_43db_a562_7eedafad93ce.slice/crio-218e009626e046835e9aaf36f349d8887fa4f9e062d4983d10391af368215be3 WatchSource:0}: Error finding container 218e009626e046835e9aaf36f349d8887fa4f9e062d4983d10391af368215be3: Status 404 returned error can't find the container with id 218e009626e046835e9aaf36f349d8887fa4f9e062d4983d10391af368215be3 Sep 17 18:29:12.824680 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.824634 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a586aaa_a2e4_462e_8c3d_617310c54a19.slice/crio-7815ee4e3f76a11f14ff7749194acf4ba2203799170f667bd5835506686b6ffd WatchSource:0}: Error finding container 7815ee4e3f76a11f14ff7749194acf4ba2203799170f667bd5835506686b6ffd: Status 404 returned error can't find the container with id 7815ee4e3f76a11f14ff7749194acf4ba2203799170f667bd5835506686b6ffd Sep 17 18:29:12.825423 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.825214 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4073a8_521e_483a_abb9_0506b859d73c.slice/crio-961a85df25885aaa813804d39d3015ff2853964efa1fe2e9ebbf2b833adad258 WatchSource:0}: Error finding container 961a85df25885aaa813804d39d3015ff2853964efa1fe2e9ebbf2b833adad258: Status 404 returned error can't find the container with id 961a85df25885aaa813804d39d3015ff2853964efa1fe2e9ebbf2b833adad258 Sep 17 18:29:12.826424 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:12.826272 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd409999b_ee91_4485_868b_4162e1b49553.slice/crio-ca090f823c16c68c47ed025404129e3603be2715dc0212b2f1dad12f7f3f4488 WatchSource:0}: Error finding container ca090f823c16c68c47ed025404129e3603be2715dc0212b2f1dad12f7f3f4488: Status 404 returned error can't find the container with id ca090f823c16c68c47ed025404129e3603be2715dc0212b2f1dad12f7f3f4488 Sep 17 18:29:12.893387 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.893259 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:11 +0000 UTC" deadline="2028-05-17 03:25:01.605029932 +0000 UTC" Sep 17 18:29:12.893387 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.893386 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14576h55m48.711646711s" Sep 17 18:29:12.986950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.986925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:12.986950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.986944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:12.987090 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.987027 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:12.987165 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:12.987148 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:12.996989 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:12.996966 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" event={"ID":"aa4163b5-6ed7-4af9-a018-c265369dda47","Type":"ContainerStarted","Data":"6cdb517a33f0476aec22e4c6ca887aa2dfac7076b1a62862a79090619ce68b85"} Sep 17 18:29:13.000352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.000325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" event={"ID":"cb7ea012701865f5588b4ef5f36a7e27","Type":"ContainerStarted","Data":"4dae048e1aa9d5688cdd05f842bb5583e30d27f62ee1051462c3e6d937419cbd"} Sep 17 18:29:13.001507 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.001477 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f2fg6" event={"ID":"5a586aaa-a2e4-462e-8c3d-617310c54a19","Type":"ContainerStarted","Data":"7815ee4e3f76a11f14ff7749194acf4ba2203799170f667bd5835506686b6ffd"} Sep 17 18:29:13.002890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.002871 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerStarted","Data":"218e009626e046835e9aaf36f349d8887fa4f9e062d4983d10391af368215be3"} Sep 17 18:29:13.004421 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.004397 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csc94" event={"ID":"2fe05e5e-1aff-454a-9409-585a30fc2aae","Type":"ContainerStarted","Data":"81da1ac21454f15d53a2da6b809aa286e9a3c412dfe0311388d9be457af96f74"} Sep 17 18:29:13.005540 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.005516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"ca090f823c16c68c47ed025404129e3603be2715dc0212b2f1dad12f7f3f4488"} Sep 17 18:29:13.006592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.006569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j2lmq" event={"ID":"4583ef38-ee5c-4056-9acd-00782d3adf4b","Type":"ContainerStarted","Data":"08a957e2bd75dac90fd89986414462290ebb536f1fa736e123e215e586040b60"} Sep 17 18:29:13.007614 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.007595 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g72sf" event={"ID":"dd4073a8-521e-483a-abb9-0506b859d73c","Type":"ContainerStarted","Data":"961a85df25885aaa813804d39d3015ff2853964efa1fe2e9ebbf2b833adad258"} Sep 17 18:29:13.008732 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.008708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-29dvp" event={"ID":"f5011467-2009-4336-ac87-cd9bf72b1881","Type":"ContainerStarted","Data":"1af3f3e012f7eee53897d1fb5a5d3ed66deb5a832a62e78c7d5201c73f7ca180"} Sep 17 18:29:13.009808 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.009789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" event={"ID":"af33fe1c-3c60-4691-b297-97e78954f284","Type":"ContainerStarted","Data":"6e97af3d1597285197c49da9c5e8401b4d936b1a75a39d34ef96f9242a78ee49"} Sep 17 18:29:13.015934 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.015900 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" podStartSLOduration=3.015882929 podStartE2EDuration="3.015882929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:13.015664682 +0000 UTC m=+5.004280931" watchObservedRunningTime="2026-09-17 18:29:13.015882929 +0000 UTC m=+5.004499178" Sep 17 18:29:13.410798 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.410717 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:13.477159 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.477125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:13.477321 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.477186 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:13.477321 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.477289 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:13.477484 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.477368 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:15.477347837 +0000 UTC m=+7.465964078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:13.477692 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.477296 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:13.477769 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.477728 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:15.477713425 +0000 UTC m=+7.466329652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:13.578543 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.578510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:13.578700 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.578660 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:13.578700 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.578677 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:13.578700 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.578688 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:13.578860 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.578740 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:15.578723683 +0000 UTC m=+7.567339970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:13.895246 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.894191 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:11 +0000 UTC" deadline="2028-05-10 04:42:30.665355603 +0000 UTC" Sep 17 18:29:13.895246 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.894236 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14410h13m16.771123572s" Sep 17 18:29:13.989714 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:13.989216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:13.989714 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:13.989332 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:14.023178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:14.023065 2559 generic.go:356] "Generic (PLEG): container finished" podID="4bc2e9e1a999339d953e286eb83e762e" containerID="17e5d260dae3958981cf24e0f4e02a56ab489452607d02ced4281a409e5b0ad7" exitCode=0 Sep 17 18:29:14.023178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:14.023106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"4bc2e9e1a999339d953e286eb83e762e","Type":"ContainerDied","Data":"17e5d260dae3958981cf24e0f4e02a56ab489452607d02ced4281a409e5b0ad7"} Sep 17 18:29:14.987302 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:14.987267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:14.987695 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:14.987408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:14.987695 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:14.987529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:14.987695 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:14.987617 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:15.044867 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.044832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"4bc2e9e1a999339d953e286eb83e762e","Type":"ContainerStarted","Data":"d12ffa1f5af376ef356b0eb5e96b2b0a3e9b251361b027f633e47c23af94ea2f"} Sep 17 18:29:15.061076 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.060665 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" podStartSLOduration=5.0606523 podStartE2EDuration="5.0606523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:15.060527865 +0000 UTC m=+7.049144118" watchObservedRunningTime="2026-09-17 18:29:15.0606523 +0000 UTC m=+7.049268549" Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.494024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.494105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.494188 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.494208 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.494265 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:19.494243907 +0000 UTC m=+11.482860161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:15.494330 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.494285 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:19.494275618 +0000 UTC m=+11.482891854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:15.595171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.595132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:15.595339 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.595295 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:15.595339 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.595317 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:15.595339 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.595331 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:15.595520 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.595387 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:19.595369648 +0000 UTC m=+11.583985889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:15.987413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:15.986903 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:15.987413 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:15.987024 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:16.987879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:16.987711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:16.987879 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:16.987859 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:16.988385 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:16.987722 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:16.988385 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:16.988287 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:17.987568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:17.987538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:17.987755 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:17.987675 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:18.987618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:18.987583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:18.988122 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:18.987712 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:18.988122 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:18.988055 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:18.988251 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:18.988190 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:19.526696 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:19.526662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:19.526882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:19.526725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:19.526882 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.526860 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:19.526992 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.526920 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.526901124 +0000 UTC m=+19.515517353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:19.527277 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.527257 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:19.527351 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.527316 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.527298894 +0000 UTC m=+19.515915122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:19.627412 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:19.627377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:19.627603 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.627590 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:19.627670 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.627615 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:19.627670 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.627628 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:19.627771 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.627681 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:27.627665 +0000 UTC m=+19.616281242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:19.988058 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:19.987968 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:19.988512 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:19.988090 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:20.988198 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:20.987504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:20.988198 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:20.987635 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:20.988198 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:20.987996 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:20.988198 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:20.988084 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:21.987187 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:21.987160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:21.987347 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:21.987261 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:22.987496 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:22.987410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:22.987496 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:22.987470 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:22.987975 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:22.987555 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:22.987975 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:22.987653 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:23.986973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:23.986938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:23.987155 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:23.987072 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:24.987568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:24.987537 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:24.987568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:24.987560 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:24.988096 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:24.987667 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:24.988096 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:24.987805 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:25.987561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:25.987529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:25.987727 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:25.987660 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:26.987831 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:26.987798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:26.987831 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:26.987839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:26.988328 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:26.987921 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:26.988328 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:26.988041 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:27.590098 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:27.590066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:27.590269 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:27.590142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:27.590269 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.590228 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:27.590269 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.590242 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:27.590435 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.590302 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret podName:ea05b683-b14d-4b78-b315-f3daaed894e0 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.590282789 +0000 UTC m=+35.578899031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret") pod "global-pull-secret-syncer-lx2z8" (UID: "ea05b683-b14d-4b78-b315-f3daaed894e0") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:27.590435 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.590321 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.590313728 +0000 UTC m=+35.578929956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:27.691383 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:27.691356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:27.691550 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.691494 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:27.691550 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.691514 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:27.691550 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.691525 2559 projected.go:196] Error preparing data for projected volume kube-api-access-lv5b2 for pod openshift-network-diagnostics/network-check-target-lx76w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:27.691665 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.691573 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2 podName:ae36918f-41a8-4fd0-abdd-42d3b478f0f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.691560285 +0000 UTC m=+35.680176512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lv5b2" (UniqueName: "kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2") pod "network-check-target-lx76w" (UID: "ae36918f-41a8-4fd0-abdd-42d3b478f0f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:27.990984 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:27.990914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:27.991366 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:27.991027 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:28.987124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:28.987094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:28.987306 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:28.987202 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:28.987306 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:28.987262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:28.987971 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:28.987711 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:29.987625 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:29.987595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:29.987925 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:29.987707 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:30.987815 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:30.987479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:30.988529 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:30.987479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:30.988529 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:30.987924 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:30.988529 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:30.988006 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:31.079200 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.079172 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f2fg6" event={"ID":"5a586aaa-a2e4-462e-8c3d-617310c54a19","Type":"ContainerStarted","Data":"3e349737dec6f802b491c70f279a86bdbcc73578a33a9c961d63b6ccfe836f0f"} Sep 17 18:29:31.080663 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.080641 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="77b5bf96c7294a3bb124ad598debefa1e2b232565d7898959ecaedd0f89eced1" exitCode=0 Sep 17 18:29:31.080764 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.080708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"77b5bf96c7294a3bb124ad598debefa1e2b232565d7898959ecaedd0f89eced1"} Sep 17 18:29:31.082179 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.082125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csc94" event={"ID":"2fe05e5e-1aff-454a-9409-585a30fc2aae","Type":"ContainerStarted","Data":"a1ff697e20fd8dbe3759619d4e645fd6a444015b1c03efd4739c69c49f08c42d"} Sep 17 18:29:31.085212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:29:31.085605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085582 2559 generic.go:356] "Generic (PLEG): container finished" podID="d409999b-ee91-4485-868b-4162e1b49553" containerID="e2959cb4d93624248fcf9e3870ad2101f3a7932f1dac524896e2c9e0dfba02fa" exitCode=1 Sep 17 18:29:31.085701 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"01a66c1c68b619c3fdd7fa27bc1d373dce7591d03b97410b3cc58fbe5fd9450b"} Sep 17 18:29:31.085758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"e1176a2a9b58573107ae7e588cfe3b957575332ac29abe17bff5dbc926f9484e"} Sep 17 18:29:31.085826 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"d55fb14460d6ffa1cd6164288f96e313b7e775f6d6322b57d4f8057f6cab2702"} Sep 17 18:29:31.085875 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"f93bdfa0a74b530fd465abbe0fa278feabd4535525f2bd82fdf8b779ac478970"} Sep 17 18:29:31.085920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerDied","Data":"e2959cb4d93624248fcf9e3870ad2101f3a7932f1dac524896e2c9e0dfba02fa"} Sep 17 18:29:31.085964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.085929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"fbca729660924b5ec4baf801591811cd481e06652814bef2bd553ac8a2d1861f"} Sep 17 18:29:31.087616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.087598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j2lmq" event={"ID":"4583ef38-ee5c-4056-9acd-00782d3adf4b","Type":"ContainerStarted","Data":"abfa8d11cd065413f4e2616d7b2ca8cb97c4529234636f73045e3892e9a37502"} Sep 17 18:29:31.089392 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.089372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g72sf" event={"ID":"dd4073a8-521e-483a-abb9-0506b859d73c","Type":"ContainerStarted","Data":"93783a34d3a434759b00422bde9eec1ef52e0b84c71e09c18858e70f52189c42"} Sep 17 18:29:31.090819 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.090797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-29dvp" event={"ID":"f5011467-2009-4336-ac87-cd9bf72b1881","Type":"ContainerStarted","Data":"94096766f3d37083a668b76a67c573d1cbcd6b2a0c2bd5a9d82574a71ad8ae8e"} Sep 17 18:29:31.092347 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.092329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" event={"ID":"af33fe1c-3c60-4691-b297-97e78954f284","Type":"ContainerStarted","Data":"e0a56e085d2bb4a6984e0166f8e57cf03eea6cce16327e081a6413ebdeb3ae50"} Sep 17 18:29:31.093879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.093855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" event={"ID":"aa4163b5-6ed7-4af9-a018-c265369dda47","Type":"ContainerStarted","Data":"baae2eda13bd129a111fc3ee9ab8da0a369ec5618de1829949317de315eaec7b"} Sep 17 18:29:31.093997 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.093957 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f2fg6" podStartSLOduration=4.096453339 podStartE2EDuration="21.093941436s" totalImagesPullingTime="16.997488097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.093706473 +0000 UTC m=+23.082322723" watchObservedRunningTime="2026-09-17 18:29:31.093941436 +0000 UTC m=+23.082557686" Sep 17 18:29:31.131776 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.131734 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g72sf" podStartSLOduration=3.9121961450000002 podStartE2EDuration="21.131715132s" totalImagesPullingTime="17.219518987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.131601222 +0000 UTC m=+23.120217480" watchObservedRunningTime="2026-09-17 18:29:31.131715132 +0000 UTC m=+23.120331387" Sep 17 18:29:31.148405 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.148358 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-29dvp" podStartSLOduration=4.150551191 podStartE2EDuration="21.14834175s" totalImagesPullingTime="16.997790559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.147698641 +0000 UTC m=+23.136314889" watchObservedRunningTime="2026-09-17 18:29:31.14834175 +0000 UTC m=+23.136958001" Sep 17 18:29:31.163220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.163184 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j2lmq" podStartSLOduration=4.00912576 podStartE2EDuration="21.163172523s" totalImagesPullingTime="17.154046763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.163033901 +0000 UTC m=+23.151650151" watchObservedRunningTime="2026-09-17 18:29:31.163172523 +0000 UTC m=+23.151788771" Sep 17 18:29:31.178562 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.178525 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-csc94" podStartSLOduration=11.369687463 podStartE2EDuration="21.178509009s" totalImagesPullingTime="9.808821546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.17828935 +0000 UTC m=+23.166905599" watchObservedRunningTime="2026-09-17 18:29:31.178509009 +0000 UTC m=+23.167125259" Sep 17 18:29:31.201631 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.201595 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gd8cq" podStartSLOduration=4.022994646 podStartE2EDuration="21.201584257s" totalImagesPullingTime="17.178589611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:31.201438126 +0000 UTC m=+23.190054380" watchObservedRunningTime="2026-09-17 18:29:31.201584257 +0000 UTC m=+23.190200505" Sep 17 18:29:31.732292 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.732171 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 18:29:31.959727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.959569 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:31.732190802Z","UUID":"9c55725e-f781-4db4-b6b4-943267ff2926","Handler":null,"Name":"","Endpoint":""} Sep 17 18:29:31.961209 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.961189 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 18:29:31.961332 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.961219 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 18:29:31.987507 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:31.987470 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:31.987641 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:31.987591 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:32.098973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:32.098883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" event={"ID":"af33fe1c-3c60-4691-b297-97e78954f284","Type":"ContainerStarted","Data":"4fe7d3b4e9703b233245e0e75e08d3ee03a1b459dcdde7032942eaee646f901c"} Sep 17 18:29:32.987149 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:32.987121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:32.987308 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:32.987154 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:32.987308 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:32.987261 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:32.987413 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:32.987338 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:33.105222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:33.105071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:29:33.105764 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:33.105737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"9f46fab8eb32862215c5fdf972f39efab5fcb1f7310f0cc53dc702d0028539f0"} Sep 17 18:29:33.326266 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:33.326238 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:33.326931 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:33.326911 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:33.987403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:33.987374 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:33.987626 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:33.987509 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:34.110381 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.110336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" event={"ID":"af33fe1c-3c60-4691-b297-97e78954f284","Type":"ContainerStarted","Data":"57869f5a3552e3bee79fae0be4adcce57e0df937d45844e6a86309f6b80760f2"} Sep 17 18:29:34.110845 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.110697 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:34.111151 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.111133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-csc94" Sep 17 18:29:34.134259 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.134199 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74lrp" podStartSLOduration=4.342055264 podStartE2EDuration="24.134185884s" totalImagesPullingTime="19.79213062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:34.13040555 +0000 UTC m=+26.119021797" watchObservedRunningTime="2026-09-17 18:29:34.134185884 +0000 UTC m=+26.122802133" Sep 17 18:29:34.987731 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.987705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:34.987865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:34.987705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:34.987865 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:34.987822 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:34.987979 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:34.987890 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:35.987388 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:35.987193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:35.988202 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:35.987417 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:36.116137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.116105 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="9ce0e36c6a6ab2d0e08e71914172d245e3372e5356ec08ced159e1f24a7dca62" exitCode=0 Sep 17 18:29:36.116224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.116180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"9ce0e36c6a6ab2d0e08e71914172d245e3372e5356ec08ced159e1f24a7dca62"} Sep 17 18:29:36.119676 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.119661 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:29:36.120047 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.120026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"d9f6e2a2eb88dda6bb2df1582f206a65027baa7d6a9bbdd0e7aaee601ddfc5eb"} Sep 17 18:29:36.120402 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.120386 2559 scope.go:117] "RemoveContainer" containerID="e2959cb4d93624248fcf9e3870ad2101f3a7932f1dac524896e2c9e0dfba02fa" Sep 17 18:29:36.120499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.120388 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:36.120554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.120526 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:36.135031 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.135014 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:36.987571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.987541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:36.987959 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:36.987693 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:36.987959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:36.987733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:36.987959 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:36.987812 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:37.126783 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.126760 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:29:37.127108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.127089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" event={"ID":"d409999b-ee91-4485-868b-4162e1b49553","Type":"ContainerStarted","Data":"d9f3c747001395364d1e4e7efd56d7033213055812abe5f08d743abc663184ea"} Sep 17 18:29:37.127376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.127355 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:37.142182 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.142163 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:29:37.163406 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.163368 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" podStartSLOduration=9.994735123 podStartE2EDuration="27.163352901s" totalImagesPullingTime="17.168617778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:37.162971253 +0000 UTC m=+29.151587503" watchObservedRunningTime="2026-09-17 18:29:37.163352901 +0000 UTC m=+29.151969151" Sep 17 18:29:37.330028 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.330000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lx2z8"] Sep 17 18:29:37.330110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.330097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:37.330236 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:37.330202 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:37.333762 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.333742 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jq4tl"] Sep 17 18:29:37.333846 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.333821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:37.333929 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:37.333910 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:37.345543 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.345524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lx76w"] Sep 17 18:29:37.345690 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:37.345613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:37.345740 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:37.345700 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:38.130658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:38.130632 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="93fecbb8df0eaff418ac3343aee4463bada4023ab5e6e549ca91b59ffe48c6cd" exitCode=0 Sep 17 18:29:38.130965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:38.130706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"93fecbb8df0eaff418ac3343aee4463bada4023ab5e6e549ca91b59ffe48c6cd"} Sep 17 18:29:38.987461 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:38.987418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:38.987634 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:38.987527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:38.987634 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:38.987534 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:38.987634 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:38.987553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:38.987795 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:38.987630 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:38.987795 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:38.987701 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:40.137705 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:40.137549 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="ab43e32e3dee9fbc25f2b5d5b875ac6468be94f4cbb8aa033760429e29cf30e5" exitCode=0 Sep 17 18:29:40.138043 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:40.137625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"ab43e32e3dee9fbc25f2b5d5b875ac6468be94f4cbb8aa033760429e29cf30e5"} Sep 17 18:29:40.987770 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:40.987732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:40.987918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:40.987862 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:40.987918 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:40.987873 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jq4tl" podUID="e08ab8d9-eb73-4356-b19c-3250af393478" Sep 17 18:29:40.987918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:40.987894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:40.988039 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:40.987983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx2z8" podUID="ea05b683-b14d-4b78-b315-f3daaed894e0" Sep 17 18:29:40.988082 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:40.988064 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lx76w" podUID="ae36918f-41a8-4fd0-abdd-42d3b478f0f3" Sep 17 18:29:42.781863 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.781836 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeReady" Sep 17 18:29:42.782514 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.781970 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 18:29:42.830746 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.830719 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:29:42.851640 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.851616 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fw9dt"] Sep 17 18:29:42.851777 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.851703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:42.854255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.854231 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 18:29:42.854347 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.854303 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 18:29:42.854674 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.854656 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 18:29:42.854849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.854823 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5mhkt\"" Sep 17 18:29:42.866027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.866007 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 18:29:42.872136 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.872109 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm"] Sep 17 18:29:42.872258 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.872239 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:42.875325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.875302 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 18:29:42.877990 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.877971 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 18:29:42.878326 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.878299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n4wfx\"" Sep 17 18:29:42.882391 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.881489 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.882391 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.881938 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.889616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.889596 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 18:29:42.890996 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.890978 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d"] Sep 17 18:29:42.891241 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.891213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:42.896144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.896122 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 18:29:42.896257 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.896190 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4fnc2\"" Sep 17 18:29:42.896257 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.896239 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 18:29:42.896554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.896533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.896627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.896607 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.911968 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.911951 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-8kmsh"] Sep 17 18:29:42.912091 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.912076 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:42.919238 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.919214 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 18:29:42.919520 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.919502 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.919664 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.919644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.919755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.919562 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tg2lx\"" Sep 17 18:29:42.919755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.919696 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 18:29:42.927736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.927712 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c"] Sep 17 18:29:42.927865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.927847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:42.937587 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.937382 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 18:29:42.937587 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.937489 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 18:29:42.937587 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.937497 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.937587 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.937529 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.937982 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.937963 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6xmvv\"" Sep 17 18:29:42.941952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.941928 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf"] Sep 17 18:29:42.942027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.942008 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:42.944045 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.944022 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 18:29:42.949552 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.949536 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 17 18:29:42.950640 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.950621 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 17 18:29:42.951236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.951220 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.956953 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.956936 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rpqfm\"" Sep 17 18:29:42.958325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.958306 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.961509 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.961492 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7748c668d8-2g2jw"] Sep 17 18:29:42.961614 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.961598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:42.964203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.964177 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8wtfv\"" Sep 17 18:29:42.964741 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.964572 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 18:29:42.966098 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.965604 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.966098 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.965825 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.984948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.984926 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-crrcl"] Sep 17 18:29:42.985054 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.985037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:42.988432 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988411 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6fcrg\"" Sep 17 18:29:42.988533 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988470 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 18:29:42.988533 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988480 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 18:29:42.988743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988723 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 18:29:42.988833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988778 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 18:29:42.988833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.988799 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 18:29:42.989086 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:42.989068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 18:29:43.000104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wxp\" (UniqueName: \"kubernetes.io/projected/38512eae-3e24-4fd5-992b-2861f70dbedc-kube-api-access-b8wxp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.000193 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000122 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3780a308-335e-49bc-9739-ffee7515f558-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.000193 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000193 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acf34cb8-1ea1-401d-877f-0e11ead44a7d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.000318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000288 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.000318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-snapshots\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.000489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9nr\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-tmp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.000489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xlx\" (UniqueName: \"kubernetes.io/projected/3780a308-335e-49bc-9739-ffee7515f558-kube-api-access-z5xlx\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxsp\" (UniqueName: \"kubernetes.io/projected/acf34cb8-1ea1-401d-877f-0e11ead44a7d-kube-api-access-cqxsp\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-trusted-ca\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8td\" (UniqueName: \"kubernetes.io/projected/4efa5014-c43a-41f9-937a-f83becc7cd54-kube-api-access-jk8td\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.000698 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-config\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.001006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4efa5014-c43a-41f9-937a-f83becc7cd54-serving-cert\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.001006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.001006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38512eae-3e24-4fd5-992b-2861f70dbedc-serving-cert\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.001006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.000847 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acf34cb8-1ea1-401d-877f-0e11ead44a7d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.006991 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.006972 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf"] Sep 17 18:29:43.007098 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.007082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.011088 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.011068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6cjgz\"" Sep 17 18:29:43.011173 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.011133 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 18:29:43.011259 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.011238 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 18:29:43.025657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.025637 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r"] Sep 17 18:29:43.025769 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.025753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" Sep 17 18:29:43.028586 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.028571 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-64dqh\"" Sep 17 18:29:43.029510 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.029490 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:43.029661 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.029646 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:43.038143 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038125 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:29:43.038244 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038152 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d"] Sep 17 18:29:43.038244 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038170 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:43.038244 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038174 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dn5p4"] Sep 17 18:29:43.038387 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038329 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:43.038387 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.038503 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.038417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:43.041077 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.041060 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:29:43.041387 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.041369 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:29:43.042027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042009 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:29:43.042027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042015 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 18:29:43.042174 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ln4zj\"" Sep 17 18:29:43.042174 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042169 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-crd74\"" Sep 17 18:29:43.042277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042187 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:29:43.042366 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042336 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vtl4h\"" Sep 17 18:29:43.042479 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.042406 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 18:29:43.059324 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059303 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm"] Sep 17 18:29:43.059505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059338 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-8kmsh"] Sep 17 18:29:43.059505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059359 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf"] Sep 17 18:29:43.059505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fw9dt"] Sep 17 18:29:43.059505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059398 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94"] Sep 17 18:29:43.059655 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.059583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.062171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.062152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 18:29:43.062962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.062944 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 18:29:43.062962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.062957 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 18:29:43.063365 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.063347 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wggns\"" Sep 17 18:29:43.074925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.074910 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crrcl"] Sep 17 18:29:43.075011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.074939 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7748c668d8-2g2jw"] Sep 17 18:29:43.075011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.074957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r"] Sep 17 18:29:43.075011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.074976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c"] Sep 17 18:29:43.075011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.075000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn5p4"] Sep 17 18:29:43.075176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.075019 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf"] Sep 17 18:29:43.075176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.075042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94"] Sep 17 18:29:43.075176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.075071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" Sep 17 18:29:43.077421 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.077403 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mbtz2\"" Sep 17 18:29:43.102109 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.102221 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-config\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.102221 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.102221 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102176 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38512eae-3e24-4fd5-992b-2861f70dbedc-serving-cert\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.102786 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wxp\" (UniqueName: \"kubernetes.io/projected/38512eae-3e24-4fd5-992b-2861f70dbedc-kube-api-access-b8wxp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.102861 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3780a308-335e-49bc-9739-ffee7515f558-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.102861 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-config\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.102954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.102954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7736fba4-d5d5-4bb2-b963-28ed95471ce9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.102954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srkb6\" (UniqueName: \"kubernetes.io/projected/d82aeee7-bb8c-4483-abf5-4e88aae4e698-kube-api-access-srkb6\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.102954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.102943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-tmp-dir\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.103140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d82aeee7-bb8c-4483-abf5-4e88aae4e698-serving-cert\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.103140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.103140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-stats-auth\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xlx\" (UniqueName: \"kubernetes.io/projected/3780a308-335e-49bc-9739-ffee7515f558-kube-api-access-z5xlx\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103261 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8td\" (UniqueName: \"kubernetes.io/projected/4efa5014-c43a-41f9-937a-f83becc7cd54-kube-api-access-jk8td\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103288 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxsp\" (UniqueName: \"kubernetes.io/projected/acf34cb8-1ea1-401d-877f-0e11ead44a7d-kube-api-access-cqxsp\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9jz\" (UniqueName: \"kubernetes.io/projected/20981ed1-ad71-47c2-85d8-9116f29ca535-kube-api-access-6l9jz\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.103378 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4efa5014-c43a-41f9-937a-f83becc7cd54-serving-cert\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acf34cb8-1ea1-401d-877f-0e11ead44a7d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acf34cb8-1ea1-401d-877f-0e11ead44a7d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-config-volume\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-snapshots\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhls5\" (UniqueName: \"kubernetes.io/projected/318bf57e-5b23-476e-9f2d-a9f9730425c0-kube-api-access-vhls5\") pod \"volume-data-source-validator-5c4db8bdf9-z87pf\" (UID: \"318bf57e-5b23-476e-9f2d-a9f9730425c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9nr\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.103680 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:43.103736 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.103731 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.603713673 +0000 UTC m=+35.592329908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:43.104066 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.103826 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:43.104066 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.103840 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:43.104066 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.103885 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.603868502 +0000 UTC m=+35.592484746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:43.104066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-tmp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-snapshots\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38512eae-3e24-4fd5-992b-2861f70dbedc-tmp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104765 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-trusted-ca\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-default-certificate\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9tt\" (UniqueName: \"kubernetes.io/projected/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-kube-api-access-hh9tt\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d82aeee7-bb8c-4483-abf5-4e88aae4e698-config\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.105080 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.104917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9jg\" (UniqueName: \"kubernetes.io/projected/e6d0f38d-71ce-4a7e-9d11-31694639677f-kube-api-access-fd9jg\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.105521 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.105104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.106846 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.106143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efa5014-c43a-41f9-937a-f83becc7cd54-trusted-ca\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.106992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.106410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acf34cb8-1ea1-401d-877f-0e11ead44a7d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.107128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.105047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.107253 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.106889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38512eae-3e24-4fd5-992b-2861f70dbedc-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.107253 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.107009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.107363 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.107276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4efa5014-c43a-41f9-937a-f83becc7cd54-serving-cert\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.107501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.107484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38512eae-3e24-4fd5-992b-2861f70dbedc-serving-cert\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.107570 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.103558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3780a308-335e-49bc-9739-ffee7515f558-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.107868 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.107847 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.110738 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.108688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acf34cb8-1ea1-401d-877f-0e11ead44a7d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.111595 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.111573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.118984 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.118962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9nr\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.121744 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.121724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.122573 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.122551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8td\" (UniqueName: \"kubernetes.io/projected/4efa5014-c43a-41f9-937a-f83becc7cd54-kube-api-access-jk8td\") pod \"console-operator-fd4bb46c-fw9dt\" (UID: \"4efa5014-c43a-41f9-937a-f83becc7cd54\") " pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.122806 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.122785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xlx\" (UniqueName: \"kubernetes.io/projected/3780a308-335e-49bc-9739-ffee7515f558-kube-api-access-z5xlx\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.123988 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.123947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxsp\" (UniqueName: \"kubernetes.io/projected/acf34cb8-1ea1-401d-877f-0e11ead44a7d-kube-api-access-cqxsp\") pod \"kube-storage-version-migrator-operator-55889f7b47-m6w5d\" (UID: \"acf34cb8-1ea1-401d-877f-0e11ead44a7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.129353 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.129326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wxp\" (UniqueName: \"kubernetes.io/projected/38512eae-3e24-4fd5-992b-2861f70dbedc-kube-api-access-b8wxp\") pod \"insights-operator-6f9c644dd9-8kmsh\" (UID: \"38512eae-3e24-4fd5-992b-2861f70dbedc\") " pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.189521 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.189439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:29:43.205853 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.205959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhls5\" (UniqueName: \"kubernetes.io/projected/318bf57e-5b23-476e-9f2d-a9f9730425c0-kube-api-access-vhls5\") pod \"volume-data-source-validator-5c4db8bdf9-z87pf\" (UID: \"318bf57e-5b23-476e-9f2d-a9f9730425c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" Sep 17 18:29:43.205959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-default-certificate\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.205959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9tt\" (UniqueName: \"kubernetes.io/projected/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-kube-api-access-hh9tt\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.205959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d82aeee7-bb8c-4483-abf5-4e88aae4e698-config\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.205978 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.205996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9jg\" (UniqueName: \"kubernetes.io/projected/e6d0f38d-71ce-4a7e-9d11-31694639677f-kube-api-access-fd9jg\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206049 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.706031142 +0000 UTC m=+35.694647386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7736fba4-d5d5-4bb2-b963-28ed95471ce9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkb6\" (UniqueName: \"kubernetes.io/projected/d82aeee7-bb8c-4483-abf5-4e88aae4e698-kube-api-access-srkb6\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.206222 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-tmp-dir\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d82aeee7-bb8c-4483-abf5-4e88aae4e698-serving-cert\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a823a4-9218-4703-ba05-6aa9d41b183b-cert\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-stats-auth\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4v6\" (UniqueName: \"kubernetes.io/projected/a2a823a4-9218-4703-ba05-6aa9d41b183b-kube-api-access-7q4v6\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxgw\" (UniqueName: \"kubernetes.io/projected/397b6ef1-7013-42ce-86e6-9f50acfaaf93-kube-api-access-sbxgw\") pod \"network-check-source-7cbc45fddc-28z94\" (UID: \"397b6ef1-7013-42ce-86e6-9f50acfaaf93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9jz\" (UniqueName: \"kubernetes.io/projected/20981ed1-ad71-47c2-85d8-9116f29ca535-kube-api-access-6l9jz\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206542 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-tmp-dir\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206546 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-config-volume\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.206656 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206660 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.706643615 +0000 UTC m=+35.695259848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206720 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206772 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.706759017 +0000 UTC m=+35.695375258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206880 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.706862939 +0000 UTC m=+35.695479191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.206955 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.206960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7736fba4-d5d5-4bb2-b963-28ed95471ce9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.207002 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.706990099 +0000 UTC m=+35.695606328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.207046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d82aeee7-bb8c-4483-abf5-4e88aae4e698-config\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.207289 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.207262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-config-volume\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.209091 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.209072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-default-certificate\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.209171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.209099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d82aeee7-bb8c-4483-abf5-4e88aae4e698-serving-cert\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.209313 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.209297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-stats-auth\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.222370 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.222347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" Sep 17 18:29:43.229547 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.229528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9jz\" (UniqueName: \"kubernetes.io/projected/20981ed1-ad71-47c2-85d8-9116f29ca535-kube-api-access-6l9jz\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.229785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.229760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhls5\" (UniqueName: \"kubernetes.io/projected/318bf57e-5b23-476e-9f2d-a9f9730425c0-kube-api-access-vhls5\") pod \"volume-data-source-validator-5c4db8bdf9-z87pf\" (UID: \"318bf57e-5b23-476e-9f2d-a9f9730425c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" Sep 17 18:29:43.230376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.230354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9tt\" (UniqueName: \"kubernetes.io/projected/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-kube-api-access-hh9tt\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.231409 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.231387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9jg\" (UniqueName: \"kubernetes.io/projected/e6d0f38d-71ce-4a7e-9d11-31694639677f-kube-api-access-fd9jg\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.237642 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.237625 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" Sep 17 18:29:43.238075 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.238049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkb6\" (UniqueName: \"kubernetes.io/projected/d82aeee7-bb8c-4483-abf5-4e88aae4e698-kube-api-access-srkb6\") pod \"service-ca-operator-57cdc4b768-xbf8c\" (UID: \"d82aeee7-bb8c-4483-abf5-4e88aae4e698\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.250969 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.250950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" Sep 17 18:29:43.307594 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.307569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a823a4-9218-4703-ba05-6aa9d41b183b-cert\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.307697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.307639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4v6\" (UniqueName: \"kubernetes.io/projected/a2a823a4-9218-4703-ba05-6aa9d41b183b-kube-api-access-7q4v6\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.307746 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.307695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxgw\" (UniqueName: \"kubernetes.io/projected/397b6ef1-7013-42ce-86e6-9f50acfaaf93-kube-api-access-sbxgw\") pod \"network-check-source-7cbc45fddc-28z94\" (UID: \"397b6ef1-7013-42ce-86e6-9f50acfaaf93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" Sep 17 18:29:43.309973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.309954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a823a4-9218-4703-ba05-6aa9d41b183b-cert\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.316523 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.316498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4v6\" (UniqueName: \"kubernetes.io/projected/a2a823a4-9218-4703-ba05-6aa9d41b183b-kube-api-access-7q4v6\") pod \"ingress-canary-dn5p4\" (UID: \"a2a823a4-9218-4703-ba05-6aa9d41b183b\") " pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.316616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.316575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxgw\" (UniqueName: \"kubernetes.io/projected/397b6ef1-7013-42ce-86e6-9f50acfaaf93-kube-api-access-sbxgw\") pod \"network-check-source-7cbc45fddc-28z94\" (UID: \"397b6ef1-7013-42ce-86e6-9f50acfaaf93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" Sep 17 18:29:43.334471 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.334438 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" Sep 17 18:29:43.375460 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.375426 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn5p4" Sep 17 18:29:43.383577 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.383558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" Sep 17 18:29:43.611105 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.611016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:43.611105 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.611076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.611143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611158 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.611197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611222 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.611203 +0000 UTC m=+36.599819245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611281 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611293 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:43.611322 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611299 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 18:29:43.611707 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611341 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.611327467 +0000 UTC m=+36.599943706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:43.611707 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.611372 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs podName:e08ab8d9-eb73-4356-b19c-3250af393478 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:15.611351258 +0000 UTC m=+67.599967502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs") pod "network-metrics-daemon-jq4tl" (UID: "e08ab8d9-eb73-4356-b19c-3250af393478") : secret "metrics-daemon-secret" not found Sep 17 18:29:43.613670 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.613650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea05b683-b14d-4b78-b315-f3daaed894e0-original-pull-secret\") pod \"global-pull-secret-syncer-lx2z8\" (UID: \"ea05b683-b14d-4b78-b315-f3daaed894e0\") " pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:43.648684 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.648659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx2z8" Sep 17 18:29:43.711866 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.711843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:43.711962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.711891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.711962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.711914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:43.711962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.711932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.711962 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.711988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712025 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.712007706 +0000 UTC m=+36.700623953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712064 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.712046112 +0000 UTC m=+36.700662359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712068 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.712094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:43.712124 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712126 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.712106264 +0000 UTC m=+36.700722508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:43.712491 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712073 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:43.712491 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712172 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:43.712491 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712197 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.712184209 +0000 UTC m=+36.700800458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:43.712491 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:43.712214 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.712204278 +0000 UTC m=+36.700820514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:43.714588 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.714570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv5b2\" (UniqueName: \"kubernetes.io/projected/ae36918f-41a8-4fd0-abdd-42d3b478f0f3-kube-api-access-lv5b2\") pod \"network-check-target-lx76w\" (UID: \"ae36918f-41a8-4fd0-abdd-42d3b478f0f3\") " pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:43.955498 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:43.955469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:44.530364 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.530051 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lx2z8"] Sep 17 18:29:44.535999 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.535962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf"] Sep 17 18:29:44.536062 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.536009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn5p4"] Sep 17 18:29:44.536902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.536887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d"] Sep 17 18:29:44.542391 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.542368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lx76w"] Sep 17 18:29:44.544916 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:44.544894 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae36918f_41a8_4fd0_abdd_42d3b478f0f3.slice/crio-dc5901dff6a3e0da63133b3837a626c323f980fc8c92c492fd7c5f760af49bf9 WatchSource:0}: Error finding container dc5901dff6a3e0da63133b3837a626c323f980fc8c92c492fd7c5f760af49bf9: Status 404 returned error can't find the container with id dc5901dff6a3e0da63133b3837a626c323f980fc8c92c492fd7c5f760af49bf9 Sep 17 18:29:44.549515 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.549431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94"] Sep 17 18:29:44.554064 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.554019 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c"] Sep 17 18:29:44.555556 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.555508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fw9dt"] Sep 17 18:29:44.555972 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:44.555865 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38512eae_3e24_4fd5_992b_2861f70dbedc.slice/crio-50a5c452c19bffd90697de1ade7ece4de79cf5eaae8b68ecc07cacad32ccf19f WatchSource:0}: Error finding container 50a5c452c19bffd90697de1ade7ece4de79cf5eaae8b68ecc07cacad32ccf19f: Status 404 returned error can't find the container with id 50a5c452c19bffd90697de1ade7ece4de79cf5eaae8b68ecc07cacad32ccf19f Sep 17 18:29:44.556575 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.556441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-8kmsh"] Sep 17 18:29:44.556831 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:29:44.556813 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397b6ef1_7013_42ce_86e6_9f50acfaaf93.slice/crio-971768cd211bb87556674811996016c516e80acf0faf399bad4619a0b6d6cdde WatchSource:0}: Error finding container 971768cd211bb87556674811996016c516e80acf0faf399bad4619a0b6d6cdde: Status 404 returned error can't find the container with id 971768cd211bb87556674811996016c516e80acf0faf399bad4619a0b6d6cdde Sep 17 18:29:44.623099 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.623068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:44.623217 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.623166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:44.623217 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.623207 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:44.623305 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.623224 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:44.623305 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.623273 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.62325447 +0000 UTC m=+38.611870709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:44.623389 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.623334 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:44.623426 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.623407 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.623387717 +0000 UTC m=+38.612003995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:44.724185 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.724159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:44.724288 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.724221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:44.724288 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.724248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:44.724386 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.724295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:44.724386 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724307 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:44.724386 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724338 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:44.724386 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724376 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.724354361 +0000 UTC m=+38.712970588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:44.724658 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724391 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:44.724658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:44.724400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:44.724658 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724422 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.724411659 +0000 UTC m=+38.713027886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:44.724658 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.724524 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:44.725319 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.725306 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.724443467 +0000 UTC m=+38.713059698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:44.725365 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.725334 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.725323971 +0000 UTC m=+38.713940200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:44.725365 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:44.725350 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:46.72534443 +0000 UTC m=+38.713960656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:45.152571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.152486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" event={"ID":"318bf57e-5b23-476e-9f2d-a9f9730425c0","Type":"ContainerStarted","Data":"569ddc9c498fd3267b6ea75cb2ae2213e812c8c835b745a28475495f7f37c4b8"} Sep 17 18:29:45.154217 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.154190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" event={"ID":"d82aeee7-bb8c-4483-abf5-4e88aae4e698","Type":"ContainerStarted","Data":"4fd785e5c58f1ded7b0176d68f60ed413677e033208f5aacb7daebc604741692"} Sep 17 18:29:45.155598 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.155547 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" event={"ID":"acf34cb8-1ea1-401d-877f-0e11ead44a7d","Type":"ContainerStarted","Data":"6bad7fdfadf5a126300c301f886149e9369e435c647819fc54ae8a9cc7cc8495"} Sep 17 18:29:45.159130 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.159013 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="c1814feaf193493e43d4962d6a601e0ae28e73821712f32f41a879e2ce0ef7f1" exitCode=0 Sep 17 18:29:45.159130 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.159112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"c1814feaf193493e43d4962d6a601e0ae28e73821712f32f41a879e2ce0ef7f1"} Sep 17 18:29:45.161013 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.160986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" event={"ID":"4efa5014-c43a-41f9-937a-f83becc7cd54","Type":"ContainerStarted","Data":"52897dc81b86d6db130e1cfc06790a04af63f1b8e9958857245b40eddf1c1de5"} Sep 17 18:29:45.162831 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.162807 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" event={"ID":"397b6ef1-7013-42ce-86e6-9f50acfaaf93","Type":"ContainerStarted","Data":"971768cd211bb87556674811996016c516e80acf0faf399bad4619a0b6d6cdde"} Sep 17 18:29:45.168212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.168130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn5p4" event={"ID":"a2a823a4-9218-4703-ba05-6aa9d41b183b","Type":"ContainerStarted","Data":"c0d166885699e910234c8902206eaa6c9e09081ce26cf5dbf8b7a7b276af1b08"} Sep 17 18:29:45.171049 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.171013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" event={"ID":"38512eae-3e24-4fd5-992b-2861f70dbedc","Type":"ContainerStarted","Data":"50a5c452c19bffd90697de1ade7ece4de79cf5eaae8b68ecc07cacad32ccf19f"} Sep 17 18:29:45.172825 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.172801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lx2z8" event={"ID":"ea05b683-b14d-4b78-b315-f3daaed894e0","Type":"ContainerStarted","Data":"30a1f9faf085499e765560d959727f6e11b125473bc074ce6e2c0fd984d89b56"} Sep 17 18:29:45.175406 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:45.175383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lx76w" event={"ID":"ae36918f-41a8-4fd0-abdd-42d3b478f0f3","Type":"ContainerStarted","Data":"dc5901dff6a3e0da63133b3837a626c323f980fc8c92c492fd7c5f760af49bf9"} Sep 17 18:29:46.197782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.196844 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c2db46f-3b5d-43db-a562-7eedafad93ce" containerID="8bed7135a57d713a38dc12acb7b8128bd77e7c0c197492de2cd9613586379f3b" exitCode=0 Sep 17 18:29:46.197782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.196927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerDied","Data":"8bed7135a57d713a38dc12acb7b8128bd77e7c0c197492de2cd9613586379f3b"} Sep 17 18:29:46.648079 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.647822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:46.648326 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.648151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:46.648326 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.648305 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:46.648467 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.648371 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.648351738 +0000 UTC m=+42.636967967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:46.648670 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.648652 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:46.648864 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.648673 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:46.648864 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.648722 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.648704739 +0000 UTC m=+42.637320990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:46.748931 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.748856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:46.748931 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.748915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:46.749137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.748967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:46.749137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.749019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:46.749137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:46.749051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:46.749301 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749173 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:46.749301 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749232 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.749212895 +0000 UTC m=+42.737829142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749549 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749641 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.749624609 +0000 UTC m=+42.738240846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749703 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749736 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.749725261 +0000 UTC m=+42.738341492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749789 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749823 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.749812255 +0000 UTC m=+42.738428486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:46.749954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:46.749884 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.749873814 +0000 UTC m=+42.738490047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:47.214272 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:47.213500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" event={"ID":"1c2db46f-3b5d-43db-a562-7eedafad93ce","Type":"ContainerStarted","Data":"5baf2e9804fc184b95a45d59d3291e68f7a8547fae46afc4383ff35ea3c375ef"} Sep 17 18:29:47.240203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:47.240150 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g9dfm" podStartSLOduration=9.132603653 podStartE2EDuration="37.240128759s" totalImagesPullingTime="28.107525106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.237710976 +0000 UTC m=+39.226327238" watchObservedRunningTime="2026-09-17 18:29:47.240128759 +0000 UTC m=+39.228745010" Sep 17 18:29:50.685882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.685844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.685920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.686021 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.686047 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.686056 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.686118 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.686096571 +0000 UTC m=+50.674712813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:50.686258 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.686135 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.686126269 +0000 UTC m=+50.674742518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:50.786498 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.786469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:50.786622 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.786519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:50.786622 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.786573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:50.786622 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.786604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:50.786622 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786612 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786668 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.786651666 +0000 UTC m=+50.775267907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786670 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786701 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786722 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.786706158 +0000 UTC m=+50.775322405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:50.786699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786740 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.786733249 +0000 UTC m=+50.775349475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786752 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786755 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.78674872 +0000 UTC m=+50.775364947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:50.786797 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:50.786777 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.786769384 +0000 UTC m=+50.775385611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:54.237952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.237184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" event={"ID":"d82aeee7-bb8c-4483-abf5-4e88aae4e698","Type":"ContainerStarted","Data":"8670dab99835e1b7ea50343fba98e62ea17f36a36af0b13d283fc24cfdd1047e"} Sep 17 18:29:54.240323 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.240299 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" event={"ID":"acf34cb8-1ea1-401d-877f-0e11ead44a7d","Type":"ContainerStarted","Data":"bfe4c9f4a847926cf8692ba055a3f654ebf7f8f8a6c504773e6652c426121faf"} Sep 17 18:29:54.243803 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.243617 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/0.log" Sep 17 18:29:54.243991 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.243780 2559 generic.go:356] "Generic (PLEG): container finished" podID="4efa5014-c43a-41f9-937a-f83becc7cd54" containerID="e261d4bec74498b2ffcc6f92dc93c8c24f6ad4786151632dac5c0e6cb483293b" exitCode=255 Sep 17 18:29:54.244284 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.244240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" event={"ID":"4efa5014-c43a-41f9-937a-f83becc7cd54","Type":"ContainerDied","Data":"e261d4bec74498b2ffcc6f92dc93c8c24f6ad4786151632dac5c0e6cb483293b"} Sep 17 18:29:54.246430 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.246039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" event={"ID":"397b6ef1-7013-42ce-86e6-9f50acfaaf93","Type":"ContainerStarted","Data":"15fcf98d93d9af264a0796d835b8a32315472b6351ec3bd9cb5a1b4aed918997"} Sep 17 18:29:54.248721 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.248300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn5p4" event={"ID":"a2a823a4-9218-4703-ba05-6aa9d41b183b","Type":"ContainerStarted","Data":"0a93227d7463b5eb8419ca1cc9c39e28c5570f351313be0c089deee74afeac0b"} Sep 17 18:29:54.250469 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.250376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" event={"ID":"38512eae-3e24-4fd5-992b-2861f70dbedc","Type":"ContainerStarted","Data":"d9520c9e5766631b266187081251abcb3b39987f748231a9e4b6f60e85d7df1c"} Sep 17 18:29:54.252699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.252431 2559 scope.go:117] "RemoveContainer" containerID="e261d4bec74498b2ffcc6f92dc93c8c24f6ad4786151632dac5c0e6cb483293b" Sep 17 18:29:54.252699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.252661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lx2z8" event={"ID":"ea05b683-b14d-4b78-b315-f3daaed894e0","Type":"ContainerStarted","Data":"63e0f1e131b31aa2ab43eef1abf11274bdf1d83dcf51d6ebb0104744054091be"} Sep 17 18:29:54.255124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.254756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lx76w" event={"ID":"ae36918f-41a8-4fd0-abdd-42d3b478f0f3","Type":"ContainerStarted","Data":"03f528e148b46ed02931a3ba7254dc0f6c257ae689aaf1e5b600b150e738e0e5"} Sep 17 18:29:54.255203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.255153 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:29:54.259901 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.259884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" event={"ID":"318bf57e-5b23-476e-9f2d-a9f9730425c0","Type":"ContainerStarted","Data":"8920c7769104e588b2d3a88331007df6ecb4ed8b82ced0513437f1721d0b0e1a"} Sep 17 18:29:54.308540 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.308492 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" podStartSLOduration=22.011098545 podStartE2EDuration="31.308476657s" totalImagesPullingTime="9.297378112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.273944315 +0000 UTC m=+46.262560565" watchObservedRunningTime="2026-09-17 18:29:54.308476657 +0000 UTC m=+46.297092908" Sep 17 18:29:54.308824 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.308783 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lx76w" podStartSLOduration=34.995770112 podStartE2EDuration="44.308775607s" totalImagesPullingTime="9.313005495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.308168744 +0000 UTC m=+46.296784995" watchObservedRunningTime="2026-09-17 18:29:54.308775607 +0000 UTC m=+46.297391853" Sep 17 18:29:54.403493 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.401510 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" podStartSLOduration=22.105106241 podStartE2EDuration="31.401493725s" totalImagesPullingTime="9.296387484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.3759925 +0000 UTC m=+46.364608749" watchObservedRunningTime="2026-09-17 18:29:54.401493725 +0000 UTC m=+46.390109975" Sep 17 18:29:54.428639 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.428584 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-z87pf" podStartSLOduration=23.24932375 podStartE2EDuration="32.428563739s" totalImagesPullingTime="9.179239989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.427134509 +0000 UTC m=+46.415750962" watchObservedRunningTime="2026-09-17 18:29:54.428563739 +0000 UTC m=+46.417179990" Sep 17 18:29:54.428763 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.428692 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lx2z8" podStartSLOduration=35.095210953 podStartE2EDuration="44.428684736s" totalImagesPullingTime="9.333473783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.402641033 +0000 UTC m=+46.391257280" watchObservedRunningTime="2026-09-17 18:29:54.428684736 +0000 UTC m=+46.417300985" Sep 17 18:29:54.450464 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.450281 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dn5p4" podStartSLOduration=3.273023972 podStartE2EDuration="12.450261684s" totalImagesPullingTime="9.177237712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.448810422 +0000 UTC m=+46.437426674" watchObservedRunningTime="2026-09-17 18:29:54.450261684 +0000 UTC m=+46.438877935" Sep 17 18:29:54.478281 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.478247 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-28z94" podStartSLOduration=22.181227699 podStartE2EDuration="31.478235498s" totalImagesPullingTime="9.297007799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.477974036 +0000 UTC m=+46.466590286" watchObservedRunningTime="2026-09-17 18:29:54.478235498 +0000 UTC m=+46.466851792" Sep 17 18:29:54.502117 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:54.502050 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" podStartSLOduration=22.185456471 podStartE2EDuration="31.50203717s" totalImagesPullingTime="9.316580699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.501331916 +0000 UTC m=+46.489948168" watchObservedRunningTime="2026-09-17 18:29:54.50203717 +0000 UTC m=+46.490653452" Sep 17 18:29:55.265531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.265502 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:29:55.265970 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.265953 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/0.log" Sep 17 18:29:55.266049 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.265995 2559 generic.go:356] "Generic (PLEG): container finished" podID="4efa5014-c43a-41f9-937a-f83becc7cd54" containerID="7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430" exitCode=255 Sep 17 18:29:55.266105 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.266070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" event={"ID":"4efa5014-c43a-41f9-937a-f83becc7cd54","Type":"ContainerDied","Data":"7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430"} Sep 17 18:29:55.266228 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.266214 2559 scope.go:117] "RemoveContainer" containerID="e261d4bec74498b2ffcc6f92dc93c8c24f6ad4786151632dac5c0e6cb483293b" Sep 17 18:29:55.266442 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:55.266423 2559 scope.go:117] "RemoveContainer" containerID="7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430" Sep 17 18:29:55.266654 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:55.266614 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fw9dt_openshift-console-operator(4efa5014-c43a-41f9-937a-f83becc7cd54)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" podUID="4efa5014-c43a-41f9-937a-f83becc7cd54" Sep 17 18:29:56.221278 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.221243 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9rvsm"] Sep 17 18:29:56.224001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.223988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.227317 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.227296 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 18:29:56.227317 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.227314 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9cvxp\"" Sep 17 18:29:56.227536 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.227299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 18:29:56.245740 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.245721 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rvsm"] Sep 17 18:29:56.271203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.271182 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:29:56.271613 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.271596 2559 scope.go:117] "RemoveContainer" containerID="7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430" Sep 17 18:29:56.271769 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.271750 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fw9dt_openshift-console-operator(4efa5014-c43a-41f9-937a-f83becc7cd54)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" podUID="4efa5014-c43a-41f9-937a-f83becc7cd54" Sep 17 18:29:56.337026 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.337007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2768270-206e-4611-9ca1-b9937f1dad79-data-volume\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.337123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.337036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4lk\" (UniqueName: \"kubernetes.io/projected/a2768270-206e-4611-9ca1-b9937f1dad79-kube-api-access-hl4lk\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.337123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.337055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2768270-206e-4611-9ca1-b9937f1dad79-crio-socket\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.337123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.337104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.337228 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.337184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2768270-206e-4611-9ca1-b9937f1dad79-data-volume\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437589 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4lk\" (UniqueName: \"kubernetes.io/projected/a2768270-206e-4611-9ca1-b9937f1dad79-kube-api-access-hl4lk\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437589 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2768270-206e-4611-9ca1-b9937f1dad79-crio-socket\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437589 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437727 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.437626 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:29:56.437727 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.437677 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.937660255 +0000 UTC m=+48.926276486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : configmap "kube-rbac-proxy" not found Sep 17 18:29:56.437727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2768270-206e-4611-9ca1-b9937f1dad79-crio-socket\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437859 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.437859 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.437838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2768270-206e-4611-9ca1-b9937f1dad79-data-volume\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.438015 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.437874 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:29:56.438015 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.437943 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.937924043 +0000 UTC m=+48.926540273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : secret "insights-runtime-extractor-tls" not found Sep 17 18:29:56.454537 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.454516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4lk\" (UniqueName: \"kubernetes.io/projected/a2768270-206e-4611-9ca1-b9937f1dad79-kube-api-access-hl4lk\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.941815 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.941789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.941933 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:56.941867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:56.941933 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.941902 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:29:56.942042 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.941956 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:29:56.942042 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.941974 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:57.941956624 +0000 UTC m=+49.930572864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : configmap "kube-rbac-proxy" not found Sep 17 18:29:56.942042 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:56.941997 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:57.941986843 +0000 UTC m=+49.930603083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : secret "insights-runtime-extractor-tls" not found Sep 17 18:29:57.949337 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:57.949314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:57.949664 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:57.949389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:57.949664 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:57.949418 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:29:57.949664 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:57.949476 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.949463633 +0000 UTC m=+51.938079874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : configmap "kube-rbac-proxy" not found Sep 17 18:29:57.949664 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:57.949476 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:29:57.949664 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:57.949521 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.949511486 +0000 UTC m=+51.938127731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : secret "insights-runtime-extractor-tls" not found Sep 17 18:29:58.374126 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.374076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j2lmq_4583ef38-ee5c-4056-9acd-00782d3adf4b/dns-node-resolver/0.log" Sep 17 18:29:58.754887 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.754864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:29:58.754985 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.754917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:29:58.755033 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.754991 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:58.755033 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.754990 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:58.755033 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.755009 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84669d5866-pk6dx: secret "image-registry-tls" not found Sep 17 18:29:58.755125 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.755035 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls podName:3780a308-335e-49bc-9739-ffee7515f558 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.755023107 +0000 UTC m=+66.743639346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4hjzm" (UID: "3780a308-335e-49bc-9739-ffee7515f558") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:58.755125 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.755047 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls podName:3b81a194-047c-45cb-a2ae-4d06c089c2f9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.755041629 +0000 UTC m=+66.743657856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls") pod "image-registry-84669d5866-pk6dx" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9") : secret "image-registry-tls" not found Sep 17 18:29:58.855149 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.855125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:29:58.855248 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.855170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:58.855248 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.855211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:29:58.855313 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855257 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:58.855313 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.855277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:29:58.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855318 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert podName:7736fba4-d5d5-4bb2-b963-28ed95471ce9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.855302483 +0000 UTC m=+66.843918724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-9nf5r" (UID: "7736fba4-d5d5-4bb2-b963-28ed95471ce9") : secret "networking-console-plugin-cert" not found Sep 17 18:29:58.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855358 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:58.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855364 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.855343157 +0000 UTC m=+66.843959387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:58.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855367 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:58.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855392 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls podName:1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.855380517 +0000 UTC m=+66.843996759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls") pod "dns-default-crrcl" (UID: "1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e") : secret "dns-default-metrics-tls" not found Sep 17 18:29:58.855570 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855413 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs podName:e6d0f38d-71ce-4a7e-9d11-31694639677f nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.855402696 +0000 UTC m=+66.844018945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs") pod "router-default-7748c668d8-2g2jw" (UID: "e6d0f38d-71ce-4a7e-9d11-31694639677f") : secret "router-metrics-certs-default" not found Sep 17 18:29:58.855570 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:58.855459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:29:58.855570 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855534 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:58.855570 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:58.855570 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls podName:20981ed1-ad71-47c2-85d8-9116f29ca535 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.855562189 +0000 UTC m=+66.844178429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d9hmf" (UID: "20981ed1-ad71-47c2-85d8-9116f29ca535") : secret "samples-operator-tls" not found Sep 17 18:29:59.374271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:59.374239 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-29dvp_f5011467-2009-4336-ac87-cd9bf72b1881/node-ca/0.log" Sep 17 18:29:59.963953 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:59.963927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:59.964097 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:59.964030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:29:59.964152 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:59.964137 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:29:59.964209 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:29:59.964190 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.964175035 +0000 UTC m=+55.952791279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : secret "insights-runtime-extractor-tls" not found Sep 17 18:29:59.964495 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:29:59.964470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2768270-206e-4611-9ca1-b9937f1dad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:30:00.175541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:00.175521 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn5p4_a2a823a4-9218-4703-ba05-6aa9d41b183b/serve-healthcheck-canary/0.log" Sep 17 18:30:00.974744 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:00.974724 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-m6w5d_acf34cb8-1ea1-401d-877f-0e11ead44a7d/kube-storage-version-migrator-operator/0.log" Sep 17 18:30:03.190226 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:03.190194 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:30:03.190618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:03.190258 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:30:03.190618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:03.190610 2559 scope.go:117] "RemoveContainer" containerID="7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430" Sep 17 18:30:03.190775 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:30:03.190754 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fw9dt_openshift-console-operator(4efa5014-c43a-41f9-937a-f83becc7cd54)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" podUID="4efa5014-c43a-41f9-937a-f83becc7cd54" Sep 17 18:30:03.993098 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:03.993067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:30:03.993255 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:30:03.993185 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:03.993255 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:30:03.993232 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls podName:a2768270-206e-4611-9ca1-b9937f1dad79 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:11.993218217 +0000 UTC m=+63.981834443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9rvsm" (UID: "a2768270-206e-4611-9ca1-b9937f1dad79") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:09.153926 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:09.153889 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gbgqb" Sep 17 18:30:12.060635 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:12.060605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:30:12.062913 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:12.062879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2768270-206e-4611-9ca1-b9937f1dad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rvsm\" (UID: \"a2768270-206e-4611-9ca1-b9937f1dad79\") " pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:30:12.136710 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:12.136680 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9cvxp\"" Sep 17 18:30:12.143271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:12.143248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rvsm" Sep 17 18:30:12.414302 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:12.414276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rvsm"] Sep 17 18:30:12.418204 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:12.418185 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2768270_206e_4611_9ca1_b9937f1dad79.slice/crio-a12bc4e59955ee31a9b209c43117f74f2000a77e2daafe6c01920202e52dfb90 WatchSource:0}: Error finding container a12bc4e59955ee31a9b209c43117f74f2000a77e2daafe6c01920202e52dfb90: Status 404 returned error can't find the container with id a12bc4e59955ee31a9b209c43117f74f2000a77e2daafe6c01920202e52dfb90 Sep 17 18:30:13.325286 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:13.325251 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rvsm" event={"ID":"a2768270-206e-4611-9ca1-b9937f1dad79","Type":"ContainerStarted","Data":"1b52ed287f2d252971f35d1ba92617765e7a6ca4185240d7918313e5a113d718"} Sep 17 18:30:13.325671 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:13.325334 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rvsm" event={"ID":"a2768270-206e-4611-9ca1-b9937f1dad79","Type":"ContainerStarted","Data":"6390dc7770b635817e5e63b342768d5420734b4a1f6829c59e002e457561536e"} Sep 17 18:30:13.325671 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:13.325352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rvsm" event={"ID":"a2768270-206e-4611-9ca1-b9937f1dad79","Type":"ContainerStarted","Data":"a12bc4e59955ee31a9b209c43117f74f2000a77e2daafe6c01920202e52dfb90"} Sep 17 18:30:14.784480 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.784427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:30:14.784861 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.784521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:30:14.787113 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.787090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"image-registry-84669d5866-pk6dx\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:30:14.787113 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.787103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3780a308-335e-49bc-9739-ffee7515f558-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4hjzm\" (UID: \"3780a308-335e-49bc-9739-ffee7515f558\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:30:14.885609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.885582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:14.885609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.885615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:14.885820 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.885670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:30:14.885820 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.885710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:30:14.885820 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.885752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:30:14.886320 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.886295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d0f38d-71ce-4a7e-9d11-31694639677f-service-ca-bundle\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:14.888416 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.888395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e-metrics-tls\") pod \"dns-default-crrcl\" (UID: \"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e\") " pod="openshift-dns/dns-default-crrcl" Sep 17 18:30:14.888758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.888711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20981ed1-ad71-47c2-85d8-9116f29ca535-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d9hmf\" (UID: \"20981ed1-ad71-47c2-85d8-9116f29ca535\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:30:14.888864 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.888841 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d0f38d-71ce-4a7e-9d11-31694639677f-metrics-certs\") pod \"router-default-7748c668d8-2g2jw\" (UID: \"e6d0f38d-71ce-4a7e-9d11-31694639677f\") " pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:14.888912 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.888885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7736fba4-d5d5-4bb2-b963-28ed95471ce9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-9nf5r\" (UID: \"7736fba4-d5d5-4bb2-b963-28ed95471ce9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:30:14.966757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.966730 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5mhkt\"" Sep 17 18:30:14.974976 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:14.974953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:30:15.005340 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.005315 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4fnc2\"" Sep 17 18:30:15.013558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.013544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" Sep 17 18:30:15.074906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.074207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8wtfv\"" Sep 17 18:30:15.082300 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.082281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" Sep 17 18:30:15.098009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.097991 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6fcrg\"" Sep 17 18:30:15.106388 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.106362 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:15.119344 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.119302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6cjgz\"" Sep 17 18:30:15.126571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.126535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crrcl" Sep 17 18:30:15.172881 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.172614 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vtl4h\"" Sep 17 18:30:15.180554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.180530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" Sep 17 18:30:15.255369 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.255343 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:30:15.257710 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:15.257686 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b81a194_047c_45cb_a2ae_4d06c089c2f9.slice/crio-bb98929bb917b1b2079d1dea73b309b41632b8063b2d7075f67e412f7b6bfddc WatchSource:0}: Error finding container bb98929bb917b1b2079d1dea73b309b41632b8063b2d7075f67e412f7b6bfddc: Status 404 returned error can't find the container with id bb98929bb917b1b2079d1dea73b309b41632b8063b2d7075f67e412f7b6bfddc Sep 17 18:30:15.290799 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.290752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm"] Sep 17 18:30:15.292884 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:15.292860 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3780a308_335e_49bc_9739_ffee7515f558.slice/crio-5c4219ca7d7b519bd93ce74521818d51178d8d18c47abc3f99aadf90c06f6a5f WatchSource:0}: Error finding container 5c4219ca7d7b519bd93ce74521818d51178d8d18c47abc3f99aadf90c06f6a5f: Status 404 returned error can't find the container with id 5c4219ca7d7b519bd93ce74521818d51178d8d18c47abc3f99aadf90c06f6a5f Sep 17 18:30:15.333500 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.333469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" event={"ID":"3780a308-335e-49bc-9739-ffee7515f558","Type":"ContainerStarted","Data":"5c4219ca7d7b519bd93ce74521818d51178d8d18c47abc3f99aadf90c06f6a5f"} Sep 17 18:30:15.335542 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.335522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rvsm" event={"ID":"a2768270-206e-4611-9ca1-b9937f1dad79","Type":"ContainerStarted","Data":"950d02d28fa1a111e4afe0450e801a95ad69c91a57f5cb2662458eb0852b4a33"} Sep 17 18:30:15.336941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.336921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" event={"ID":"3b81a194-047c-45cb-a2ae-4d06c089c2f9","Type":"ContainerStarted","Data":"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7"} Sep 17 18:30:15.337038 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.336952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" event={"ID":"3b81a194-047c-45cb-a2ae-4d06c089c2f9","Type":"ContainerStarted","Data":"bb98929bb917b1b2079d1dea73b309b41632b8063b2d7075f67e412f7b6bfddc"} Sep 17 18:30:15.337038 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.337033 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:30:15.361374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.361333 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf"] Sep 17 18:30:15.385652 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.385595 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9rvsm" podStartSLOduration=16.89211018 podStartE2EDuration="19.385576756s" totalImagesPullingTime="2.493466576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:15.365137924 +0000 UTC m=+67.353754174" watchObservedRunningTime="2026-09-17 18:30:15.385576756 +0000 UTC m=+67.374193008" Sep 17 18:30:15.386728 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.386710 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7748c668d8-2g2jw"] Sep 17 18:30:15.388281 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.388245 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" podStartSLOduration=65.3882334 podStartE2EDuration="1m5.3882334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:15.387882144 +0000 UTC m=+67.376498392" watchObservedRunningTime="2026-09-17 18:30:15.3882334 +0000 UTC m=+67.376849648" Sep 17 18:30:15.389194 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:15.389167 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d0f38d_71ce_4a7e_9d11_31694639677f.slice/crio-01b3ea10cddf9d3c78ff9b60128bfd9193d91125651aaadbab6911c755abfd61 WatchSource:0}: Error finding container 01b3ea10cddf9d3c78ff9b60128bfd9193d91125651aaadbab6911c755abfd61: Status 404 returned error can't find the container with id 01b3ea10cddf9d3c78ff9b60128bfd9193d91125651aaadbab6911c755abfd61 Sep 17 18:30:15.406287 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.406266 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crrcl"] Sep 17 18:30:15.408570 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:15.408550 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca44f7c_8d5b_4a4e_b842_d17e698d2c5e.slice/crio-f0dadc85ca31c8b744579c5ac45f5c1ca807952bb5776c17ae25d2e776c53497 WatchSource:0}: Error finding container f0dadc85ca31c8b744579c5ac45f5c1ca807952bb5776c17ae25d2e776c53497: Status 404 returned error can't find the container with id f0dadc85ca31c8b744579c5ac45f5c1ca807952bb5776c17ae25d2e776c53497 Sep 17 18:30:15.450209 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.450184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r"] Sep 17 18:30:15.454383 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:15.454349 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7736fba4_d5d5_4bb2_b963_28ed95471ce9.slice/crio-067e71a9a8097703456e0c4524038c96d79ddb6536b61572ddac1453c6ee09f3 WatchSource:0}: Error finding container 067e71a9a8097703456e0c4524038c96d79ddb6536b61572ddac1453c6ee09f3: Status 404 returned error can't find the container with id 067e71a9a8097703456e0c4524038c96d79ddb6536b61572ddac1453c6ee09f3 Sep 17 18:30:15.691225 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.691190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:30:15.693601 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.693567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08ab8d9-eb73-4356-b19c-3250af393478-metrics-certs\") pod \"network-metrics-daemon-jq4tl\" (UID: \"e08ab8d9-eb73-4356-b19c-3250af393478\") " pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:30:15.765018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.764984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-crd74\"" Sep 17 18:30:15.773208 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:15.773183 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jq4tl" Sep 17 18:30:16.052214 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.052061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jq4tl"] Sep 17 18:30:16.056565 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:16.056420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08ab8d9_eb73_4356_b19c_3250af393478.slice/crio-a62455b13bdd925897531efce7c3484c9d880faeb656cbd401572fe35906bbe7 WatchSource:0}: Error finding container a62455b13bdd925897531efce7c3484c9d880faeb656cbd401572fe35906bbe7: Status 404 returned error can't find the container with id a62455b13bdd925897531efce7c3484c9d880faeb656cbd401572fe35906bbe7 Sep 17 18:30:16.343916 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.343790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" event={"ID":"7736fba4-d5d5-4bb2-b963-28ed95471ce9","Type":"ContainerStarted","Data":"067e71a9a8097703456e0c4524038c96d79ddb6536b61572ddac1453c6ee09f3"} Sep 17 18:30:16.346186 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.346151 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7748c668d8-2g2jw" event={"ID":"e6d0f38d-71ce-4a7e-9d11-31694639677f","Type":"ContainerStarted","Data":"6647d0bff8dc088828a4bf4f339fae095fa0f3b3fba7b76d8c5c1bd4b9097ded"} Sep 17 18:30:16.346319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.346208 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7748c668d8-2g2jw" event={"ID":"e6d0f38d-71ce-4a7e-9d11-31694639677f","Type":"ContainerStarted","Data":"01b3ea10cddf9d3c78ff9b60128bfd9193d91125651aaadbab6911c755abfd61"} Sep 17 18:30:16.348089 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.348010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jq4tl" event={"ID":"e08ab8d9-eb73-4356-b19c-3250af393478","Type":"ContainerStarted","Data":"a62455b13bdd925897531efce7c3484c9d880faeb656cbd401572fe35906bbe7"} Sep 17 18:30:16.349574 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.349519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crrcl" event={"ID":"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e","Type":"ContainerStarted","Data":"f0dadc85ca31c8b744579c5ac45f5c1ca807952bb5776c17ae25d2e776c53497"} Sep 17 18:30:16.351619 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.351067 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" event={"ID":"20981ed1-ad71-47c2-85d8-9116f29ca535","Type":"ContainerStarted","Data":"6442f3e1c13b907d0cc81c0dc3b9325fb114f50a58f97233914c44406590f55c"} Sep 17 18:30:16.369577 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:16.369483 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7748c668d8-2g2jw" podStartSLOduration=53.369463693 podStartE2EDuration="53.369463693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:16.367810022 +0000 UTC m=+68.356426271" watchObservedRunningTime="2026-09-17 18:30:16.369463693 +0000 UTC m=+68.358079934" Sep 17 18:30:17.106759 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:17.106725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:17.109788 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:17.109763 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:17.355719 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:17.355687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:17.357097 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:17.357012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7748c668d8-2g2jw" Sep 17 18:30:17.988082 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:17.987980 2559 scope.go:117] "RemoveContainer" containerID="7eb0f5c1c4324eed3a8e32664f0b8c855ca9054e4b059c165a5e2625941d8430" Sep 17 18:30:18.363722 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.363650 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:30:18.364154 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.363808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" event={"ID":"4efa5014-c43a-41f9-937a-f83becc7cd54","Type":"ContainerStarted","Data":"c58c6a59bc02b1603eb45613f56b57a79a28a37d0a7a31f2c16c59a619136fd5"} Sep 17 18:30:18.364231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.364191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:30:18.365789 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.365762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" event={"ID":"20981ed1-ad71-47c2-85d8-9116f29ca535","Type":"ContainerStarted","Data":"08f3971c41f2a9c33e239b088762c31ce03733b1d259990ecb3bd80921d269c5"} Sep 17 18:30:18.384443 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.384398 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" podStartSLOduration=47.087903695 podStartE2EDuration="56.384379838s" totalImagesPullingTime="9.296476143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:18.382939409 +0000 UTC m=+70.371555657" watchObservedRunningTime="2026-09-17 18:30:18.384379838 +0000 UTC m=+70.372996087" Sep 17 18:30:18.782560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:18.782532 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-fw9dt" Sep 17 18:30:19.371557 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.371528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jq4tl" event={"ID":"e08ab8d9-eb73-4356-b19c-3250af393478","Type":"ContainerStarted","Data":"6f978e5d67703544a32c36938a877dde6614986af0bc11e9ed313cd7a6e1bd25"} Sep 17 18:30:19.371938 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.371577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jq4tl" event={"ID":"e08ab8d9-eb73-4356-b19c-3250af393478","Type":"ContainerStarted","Data":"80eb3e42fbb917ddc87dee5d6915c8ed5a9f0a22a7c71b66bf1bfd59d59984ee"} Sep 17 18:30:19.373432 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.373410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crrcl" event={"ID":"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e","Type":"ContainerStarted","Data":"86deee0ce9162abd5004fea895a9d925d6efc253dc6a70ca54b35afbcb8b30ed"} Sep 17 18:30:19.373509 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.373472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crrcl" event={"ID":"1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e","Type":"ContainerStarted","Data":"76421460dea60eae79b550355ed37d3a03373663da8a68bd84986bfe5415ca0a"} Sep 17 18:30:19.373509 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.373499 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-crrcl" Sep 17 18:30:19.375213 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.375179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" event={"ID":"20981ed1-ad71-47c2-85d8-9116f29ca535","Type":"ContainerStarted","Data":"12785792b8f0e440454d256d61b65c0cdfd215061ad5a56523a9a17b4fb26194"} Sep 17 18:30:19.376656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.376630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" event={"ID":"3780a308-335e-49bc-9739-ffee7515f558","Type":"ContainerStarted","Data":"4a410cc15976d085952424aab14631deff8550420e77682781700ff6f8c0a0d9"} Sep 17 18:30:19.377882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.377860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" event={"ID":"7736fba4-d5d5-4bb2-b963-28ed95471ce9","Type":"ContainerStarted","Data":"c6da53551c2e0bc5ef7748c7833ba05e30f984a1ffeea5ed8df9397446483def"} Sep 17 18:30:19.392231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.392192 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jq4tl" podStartSLOduration=66.997521146 podStartE2EDuration="1m9.392179473s" totalImagesPullingTime="2.394658327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.390903468 +0000 UTC m=+71.379519718" watchObservedRunningTime="2026-09-17 18:30:19.392179473 +0000 UTC m=+71.380795721" Sep 17 18:30:19.416943 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.416900 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d9hmf" podStartSLOduration=53.661544859 podStartE2EDuration="56.416885352s" totalImagesPullingTime="2.755340493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.415971 +0000 UTC m=+71.404587260" watchObservedRunningTime="2026-09-17 18:30:19.416885352 +0000 UTC m=+71.405501600" Sep 17 18:30:19.435703 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.435664 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4hjzm" podStartSLOduration=53.558254659 podStartE2EDuration="56.435652125s" totalImagesPullingTime="2.877397466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.434393864 +0000 UTC m=+71.423010113" watchObservedRunningTime="2026-09-17 18:30:19.435652125 +0000 UTC m=+71.424268373" Sep 17 18:30:19.453175 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.453139 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-crrcl" podStartSLOduration=34.682326517999996 podStartE2EDuration="37.453126521s" totalImagesPullingTime="2.770800003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.451723786 +0000 UTC m=+71.440340033" watchObservedRunningTime="2026-09-17 18:30:19.453126521 +0000 UTC m=+71.441742775" Sep 17 18:30:19.474725 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:19.474688 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-9nf5r" podStartSLOduration=53.77694606 podStartE2EDuration="56.474676287s" totalImagesPullingTime="2.697730227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.47298748 +0000 UTC m=+71.461603729" watchObservedRunningTime="2026-09-17 18:30:19.474676287 +0000 UTC m=+71.463292534" Sep 17 18:30:22.514347 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.514316 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm"] Sep 17 18:30:22.538758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.538735 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-jfk46"] Sep 17 18:30:22.538888 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.538875 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:22.542315 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.542296 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h9n7d\"" Sep 17 18:30:22.542507 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.542394 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 18:30:22.552205 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.552158 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm"] Sep 17 18:30:22.552311 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.552294 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:22.552924 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.552906 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-jfk46"] Sep 17 18:30:22.555593 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.555472 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 18:30:22.555593 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.555508 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tbsjr\"" Sep 17 18:30:22.555760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.555653 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 18:30:22.646736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.646703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdjf\" (UniqueName: \"kubernetes.io/projected/5e625bb8-3370-4be6-881a-e29b096e30d6-kube-api-access-9gdjf\") pod \"downloads-6594f4c75-jfk46\" (UID: \"5e625bb8-3370-4be6-881a-e29b096e30d6\") " pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:22.646884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.646746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10471f87-84c2-49bf-bdb1-e13094bb018c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lgtcm\" (UID: \"10471f87-84c2-49bf-bdb1-e13094bb018c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:22.747090 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.747021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10471f87-84c2-49bf-bdb1-e13094bb018c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lgtcm\" (UID: \"10471f87-84c2-49bf-bdb1-e13094bb018c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:22.747090 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.747070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdjf\" (UniqueName: \"kubernetes.io/projected/5e625bb8-3370-4be6-881a-e29b096e30d6-kube-api-access-9gdjf\") pod \"downloads-6594f4c75-jfk46\" (UID: \"5e625bb8-3370-4be6-881a-e29b096e30d6\") " pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:22.749395 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.749357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/10471f87-84c2-49bf-bdb1-e13094bb018c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lgtcm\" (UID: \"10471f87-84c2-49bf-bdb1-e13094bb018c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:22.769219 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.769188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdjf\" (UniqueName: \"kubernetes.io/projected/5e625bb8-3370-4be6-881a-e29b096e30d6-kube-api-access-9gdjf\") pod \"downloads-6594f4c75-jfk46\" (UID: \"5e625bb8-3370-4be6-881a-e29b096e30d6\") " pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:22.848220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.848190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:22.860902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:22.860875 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:23.143637 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:23.143609 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm"] Sep 17 18:30:23.146930 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:23.146899 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10471f87_84c2_49bf_bdb1_e13094bb018c.slice/crio-c2a2b4f625eccb66a5de253631c3370153cf0683fa4e1898537a804978cfbcd1 WatchSource:0}: Error finding container c2a2b4f625eccb66a5de253631c3370153cf0683fa4e1898537a804978cfbcd1: Status 404 returned error can't find the container with id c2a2b4f625eccb66a5de253631c3370153cf0683fa4e1898537a804978cfbcd1 Sep 17 18:30:23.164651 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:23.164628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-jfk46"] Sep 17 18:30:23.167654 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:23.167626 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e625bb8_3370_4be6_881a_e29b096e30d6.slice/crio-ba31ae530bcf8c7f7a76dac7714d940e8a494e3463bec83346feb9aa30ce4069 WatchSource:0}: Error finding container ba31ae530bcf8c7f7a76dac7714d940e8a494e3463bec83346feb9aa30ce4069: Status 404 returned error can't find the container with id ba31ae530bcf8c7f7a76dac7714d940e8a494e3463bec83346feb9aa30ce4069 Sep 17 18:30:23.392097 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:23.392062 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-jfk46" event={"ID":"5e625bb8-3370-4be6-881a-e29b096e30d6","Type":"ContainerStarted","Data":"ba31ae530bcf8c7f7a76dac7714d940e8a494e3463bec83346feb9aa30ce4069"} Sep 17 18:30:23.393285 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:23.393258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" event={"ID":"10471f87-84c2-49bf-bdb1-e13094bb018c","Type":"ContainerStarted","Data":"c2a2b4f625eccb66a5de253631c3370153cf0683fa4e1898537a804978cfbcd1"} Sep 17 18:30:25.404744 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:25.404700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" event={"ID":"10471f87-84c2-49bf-bdb1-e13094bb018c","Type":"ContainerStarted","Data":"48d2ab7f4c745b7f4532bdabe2e1d97be0446b2aab148ef1c32392581252b2a9"} Sep 17 18:30:25.405300 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:25.404908 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:25.410231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:25.410202 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" Sep 17 18:30:25.423382 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:25.423336 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lgtcm" podStartSLOduration=1.599509048 podStartE2EDuration="3.423322346s" totalImagesPullingTime="1.823813298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:25.421158352 +0000 UTC m=+77.409774603" watchObservedRunningTime="2026-09-17 18:30:25.423322346 +0000 UTC m=+77.411938613" Sep 17 18:30:26.273781 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:26.273749 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lx76w" Sep 17 18:30:29.383312 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:29.383284 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-crrcl" Sep 17 18:30:30.580695 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.580658 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rcnv7"] Sep 17 18:30:30.584318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.584295 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.587686 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.587661 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 18:30:30.587814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.587736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 18:30:30.587923 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.587882 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6gdgr\"" Sep 17 18:30:30.587923 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.587891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 18:30:30.588676 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.588658 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 18:30:30.711177 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-sys\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-wtmp\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-metrics-client-ca\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711274 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8d7m\" (UniqueName: \"kubernetes.io/projected/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-kube-api-access-j8d7m\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-textfile\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-root\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.711608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.711525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.812884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.812852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8d7m\" (UniqueName: \"kubernetes.io/projected/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-kube-api-access-j8d7m\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.812890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.812908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-textfile\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.812951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-root\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.812984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-root\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:30:30.813099 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-sys\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:30:30.813156 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls podName:4e5fecc6-e44f-4940-8f36-7e0ddd531b91 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.313135941 +0000 UTC m=+83.301752191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls") pod "node-exporter-rcnv7" (UID: "4e5fecc6-e44f-4940-8f36-7e0ddd531b91") : secret "node-exporter-tls" not found Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-sys\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813186 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-wtmp\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-metrics-client-ca\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813649 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-wtmp\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813649 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-textfile\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813914 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.813997 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.813976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-metrics-client-ca\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.815361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.815341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:30.823638 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:30.823611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8d7m\" (UniqueName: \"kubernetes.io/projected/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-kube-api-access-j8d7m\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:31.317318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:31.317284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:31.319992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:31.319921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5fecc6-e44f-4940-8f36-7e0ddd531b91-node-exporter-tls\") pod \"node-exporter-rcnv7\" (UID: \"4e5fecc6-e44f-4940-8f36-7e0ddd531b91\") " pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:31.495492 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:31.495467 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcnv7" Sep 17 18:30:31.507872 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:31.507836 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5fecc6_e44f_4940_8f36_7e0ddd531b91.slice/crio-fc5034785744abc77250d637dc644459c7634480d66e303f107d1c981e7347bc WatchSource:0}: Error finding container fc5034785744abc77250d637dc644459c7634480d66e303f107d1c981e7347bc: Status 404 returned error can't find the container with id fc5034785744abc77250d637dc644459c7634480d66e303f107d1c981e7347bc Sep 17 18:30:32.434929 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:32.434893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcnv7" event={"ID":"4e5fecc6-e44f-4940-8f36-7e0ddd531b91","Type":"ContainerStarted","Data":"fc5034785744abc77250d637dc644459c7634480d66e303f107d1c981e7347bc"} Sep 17 18:30:33.441624 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:33.441590 2559 generic.go:356] "Generic (PLEG): container finished" podID="4e5fecc6-e44f-4940-8f36-7e0ddd531b91" containerID="bff2cdac00b6d6899b21dccc0fac9b10564e5f603e30fc7b5b0fa181fb30f976" exitCode=0 Sep 17 18:30:33.441960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:33.441687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcnv7" event={"ID":"4e5fecc6-e44f-4940-8f36-7e0ddd531b91","Type":"ContainerDied","Data":"bff2cdac00b6d6899b21dccc0fac9b10564e5f603e30fc7b5b0fa181fb30f976"} Sep 17 18:30:34.448399 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:34.448363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcnv7" event={"ID":"4e5fecc6-e44f-4940-8f36-7e0ddd531b91","Type":"ContainerStarted","Data":"6cdcb40b3f21c0d48252ca1d8886c9844080b6ac9bbd7668972dfc06d4de4236"} Sep 17 18:30:34.448810 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:34.448417 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcnv7" event={"ID":"4e5fecc6-e44f-4940-8f36-7e0ddd531b91","Type":"ContainerStarted","Data":"c9b0b596fd57addddf3f8af2b54d6b52f0c62dbb5f25020be747aafac7030294"} Sep 17 18:30:34.469381 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:34.469339 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rcnv7" podStartSLOduration=2.6463527510000002 podStartE2EDuration="4.469324276s" totalImagesPullingTime="1.747841562s" totalInitContainerRuntime="75.129963ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:34.468133439 +0000 UTC m=+86.456749690" watchObservedRunningTime="2026-09-17 18:30:34.469324276 +0000 UTC m=+86.457940524" Sep 17 18:30:34.981690 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:34.981656 2559 patch_prober.go:28] interesting pod/image-registry-84669d5866-pk6dx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 18:30:34.981860 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:34.981713 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:30:36.356457 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.356413 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:30:36.803641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.803607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:36.807489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.807463 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.810723 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.810649 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:30:36.810723 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.810701 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:30:36.810957 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.810799 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bv2gf\"" Sep 17 18:30:36.811767 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.811748 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:30:36.812005 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.811965 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:30:36.812176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812153 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:30:36.812370 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812355 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:30:36.812423 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812385 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812626 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812670 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812769 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812849 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812926 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a74sm9j9avt6b\"" Sep 17 18:30:36.812962 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.812954 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:30:36.813374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.813040 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:30:36.813374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.813064 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:30:36.814557 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.814540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:30:36.830911 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.830875 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:36.864175 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864500 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmslt\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864500 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864500 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864443 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864500 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.864708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.865001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.864958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.965825 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.965825 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslt\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.965985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.966318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.966304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.968952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.967567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.968952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.968041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.968952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.968624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.969106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.969106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.970009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.970009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.970009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.970009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.969969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.971334 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.970730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.971334 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.970861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.971334 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.971290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.971583 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.971563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.972470 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.972431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.973106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.973053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.973205 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.973125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.973205 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.973175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.973383 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.973348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.974197 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.974178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:36.978061 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:36.978041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmslt\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt\") pod \"prometheus-k8s-0\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:37.120823 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:37.120741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:41.196791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.196761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:41.199422 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:30:41.199389 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1e4866_6131_4258_a046_06f123decc1e.slice/crio-5ca12c78f8c09a671f29e6e954f3b77883011d201a5a5c774f9edaf455edb6ac WatchSource:0}: Error finding container 5ca12c78f8c09a671f29e6e954f3b77883011d201a5a5c774f9edaf455edb6ac: Status 404 returned error can't find the container with id 5ca12c78f8c09a671f29e6e954f3b77883011d201a5a5c774f9edaf455edb6ac Sep 17 18:30:41.485034 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.484949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"5ca12c78f8c09a671f29e6e954f3b77883011d201a5a5c774f9edaf455edb6ac"} Sep 17 18:30:41.486833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.486801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-jfk46" event={"ID":"5e625bb8-3370-4be6-881a-e29b096e30d6","Type":"ContainerStarted","Data":"6d2f7088da438b064e231cb239dc6849420b70e85877cbf8c2389e2557507673"} Sep 17 18:30:41.487148 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.487132 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:41.502733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.502713 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-jfk46" Sep 17 18:30:41.509371 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:41.509323 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-jfk46" podStartSLOduration=1.704291118 podStartE2EDuration="19.509306415s" totalImagesPullingTime="17.805015297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.508101266 +0000 UTC m=+93.496717526" watchObservedRunningTime="2026-09-17 18:30:41.509306415 +0000 UTC m=+93.497922665" Sep 17 18:30:43.502664 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:43.502623 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" exitCode=0 Sep 17 18:30:43.503110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:43.502711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} Sep 17 18:30:44.679168 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:44.678478 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:30:49.533974 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:49.533934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} Sep 17 18:30:49.534476 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:49.534022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} Sep 17 18:30:51.545980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:51.545908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} Sep 17 18:30:51.546325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:51.546010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} Sep 17 18:30:51.546325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:51.546054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} Sep 17 18:30:51.546325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:51.546096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerStarted","Data":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} Sep 17 18:30:51.580773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:51.580723 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.138750966 podStartE2EDuration="15.580703775s" totalImagesPullingTime="8.421648588s" totalInitContainerRuntime="20.304221ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:51.578807039 +0000 UTC m=+103.567423287" watchObservedRunningTime="2026-09-17 18:30:51.580703775 +0000 UTC m=+103.569320028" Sep 17 18:30:52.121107 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:30:52.121084 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:00.582979 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:00.582948 2559 generic.go:356] "Generic (PLEG): container finished" podID="38512eae-3e24-4fd5-992b-2861f70dbedc" containerID="d9520c9e5766631b266187081251abcb3b39987f748231a9e4b6f60e85d7df1c" exitCode=0 Sep 17 18:31:00.583354 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:00.582985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" event={"ID":"38512eae-3e24-4fd5-992b-2861f70dbedc","Type":"ContainerDied","Data":"d9520c9e5766631b266187081251abcb3b39987f748231a9e4b6f60e85d7df1c"} Sep 17 18:31:00.583354 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:00.583264 2559 scope.go:117] "RemoveContainer" containerID="d9520c9e5766631b266187081251abcb3b39987f748231a9e4b6f60e85d7df1c" Sep 17 18:31:01.156700 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:01.156668 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7748c668d8-2g2jw_e6d0f38d-71ce-4a7e-9d11-31694639677f/router/0.log" Sep 17 18:31:01.162521 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:01.162500 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn5p4_a2a823a4-9218-4703-ba05-6aa9d41b183b/serve-healthcheck-canary/0.log" Sep 17 18:31:01.589067 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:01.588993 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-8kmsh" event={"ID":"38512eae-3e24-4fd5-992b-2861f70dbedc","Type":"ContainerStarted","Data":"8074356bb466f06ccc4e2523ccbb4eed88e25dbee3cf4f7b55d5f2a208c6f1d6"} Sep 17 18:31:04.602994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:04.602966 2559 generic.go:356] "Generic (PLEG): container finished" podID="acf34cb8-1ea1-401d-877f-0e11ead44a7d" containerID="bfe4c9f4a847926cf8692ba055a3f654ebf7f8f8a6c504773e6652c426121faf" exitCode=0 Sep 17 18:31:04.603290 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:04.603015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" event={"ID":"acf34cb8-1ea1-401d-877f-0e11ead44a7d","Type":"ContainerDied","Data":"bfe4c9f4a847926cf8692ba055a3f654ebf7f8f8a6c504773e6652c426121faf"} Sep 17 18:31:04.603358 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:04.603340 2559 scope.go:117] "RemoveContainer" containerID="bfe4c9f4a847926cf8692ba055a3f654ebf7f8f8a6c504773e6652c426121faf" Sep 17 18:31:05.609810 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:05.609774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-m6w5d" event={"ID":"acf34cb8-1ea1-401d-877f-0e11ead44a7d","Type":"ContainerStarted","Data":"29b4e2fbeb1aef460e1fb5fc47492e807f33d2997c617d9221462aff0216b48c"} Sep 17 18:31:09.702931 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:09.702858 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerName="registry" containerID="cri-o://57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7" gracePeriod=30 Sep 17 18:31:09.933257 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:09.933233 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:31:10.027385 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027385 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027361 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027392 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027409 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027557 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd9nr\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027585 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.027841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.027625 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration\") pod \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\" (UID: \"3b81a194-047c-45cb-a2ae-4d06c089c2f9\") " Sep 17 18:31:10.028032 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.028009 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:10.028472 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.028431 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:10.030108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.030081 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr" (OuterVolumeSpecName: "kube-api-access-wd9nr") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "kube-api-access-wd9nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:10.030203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.030118 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:10.030203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.030117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:10.030297 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.030275 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:10.030428 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.030412 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:10.035817 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.035795 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3b81a194-047c-45cb-a2ae-4d06c089c2f9" (UID: "3b81a194-047c-45cb-a2ae-4d06c089c2f9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:10.128489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128471 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-trusted-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128491 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-certificates\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128502 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-bound-sa-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128510 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9nr\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-kube-api-access-wd9nr\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128518 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-installation-pull-secrets\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128527 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b81a194-047c-45cb-a2ae-4d06c089c2f9-image-registry-private-configuration\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128536 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b81a194-047c-45cb-a2ae-4d06c089c2f9-registry-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.128569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.128544 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b81a194-047c-45cb-a2ae-4d06c089c2f9-ca-trust-extracted\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:10.629716 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.629691 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerID="57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7" exitCode=0 Sep 17 18:31:10.629833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.629735 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" Sep 17 18:31:10.629833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.629770 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" event={"ID":"3b81a194-047c-45cb-a2ae-4d06c089c2f9","Type":"ContainerDied","Data":"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7"} Sep 17 18:31:10.629833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.629822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84669d5866-pk6dx" event={"ID":"3b81a194-047c-45cb-a2ae-4d06c089c2f9","Type":"ContainerDied","Data":"bb98929bb917b1b2079d1dea73b309b41632b8063b2d7075f67e412f7b6bfddc"} Sep 17 18:31:10.629998 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.629857 2559 scope.go:117] "RemoveContainer" containerID="57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7" Sep 17 18:31:10.639491 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.639469 2559 scope.go:117] "RemoveContainer" containerID="57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7" Sep 17 18:31:10.640344 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:10.640316 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7\": container with ID starting with 57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7 not found: ID does not exist" containerID="57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7" Sep 17 18:31:10.640432 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.640365 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7"} err="failed to get container status \"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7\": rpc error: code = NotFound desc = could not find container \"57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7\": container with ID starting with 57b5615502e0315cfa3f407b70e46edabcc6ca8245c632ebdb464b296224a0d7 not found: ID does not exist" Sep 17 18:31:10.652630 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.652610 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:31:10.656931 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:10.656911 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84669d5866-pk6dx"] Sep 17 18:31:11.993049 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:11.993016 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" path="/var/lib/kubelet/pods/3b81a194-047c-45cb-a2ae-4d06c089c2f9/volumes" Sep 17 18:31:20.675541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:20.675513 2559 generic.go:356] "Generic (PLEG): container finished" podID="d82aeee7-bb8c-4483-abf5-4e88aae4e698" containerID="8670dab99835e1b7ea50343fba98e62ea17f36a36af0b13d283fc24cfdd1047e" exitCode=0 Sep 17 18:31:20.675856 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:20.675594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" event={"ID":"d82aeee7-bb8c-4483-abf5-4e88aae4e698","Type":"ContainerDied","Data":"8670dab99835e1b7ea50343fba98e62ea17f36a36af0b13d283fc24cfdd1047e"} Sep 17 18:31:20.675918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:20.675866 2559 scope.go:117] "RemoveContainer" containerID="8670dab99835e1b7ea50343fba98e62ea17f36a36af0b13d283fc24cfdd1047e" Sep 17 18:31:21.682176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:21.682143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-xbf8c" event={"ID":"d82aeee7-bb8c-4483-abf5-4e88aae4e698","Type":"ContainerStarted","Data":"8038cead8c607dcc63fe26ae870be7105b421f7f4380f6cbba8c99e2416032c2"} Sep 17 18:31:37.121657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:37.121613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:37.138104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:37.138080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:37.763916 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:37.763887 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.177211 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177135 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:55.177697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177464 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="prometheus" containerID="cri-o://5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" gracePeriod=600 Sep 17 18:31:55.177697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177481 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" gracePeriod=600 Sep 17 18:31:55.177697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177523 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="config-reloader" containerID="cri-o://f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" gracePeriod=600 Sep 17 18:31:55.177697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177464 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy" containerID="cri-o://2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" gracePeriod=600 Sep 17 18:31:55.177697 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177544 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-web" containerID="cri-o://4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" gracePeriod=600 Sep 17 18:31:55.177964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.177709 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="thanos-sidecar" containerID="cri-o://4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" gracePeriod=600 Sep 17 18:31:55.409606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.409578 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.577983 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.577903 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.577983 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.577960 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.577983 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.577981 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578002 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578020 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578040 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578097 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578124 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578153 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578184 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmslt\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578266 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578303 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578367 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578384 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578403 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config\") pod \"ce1e4866-6131-4258-a046-06f123decc1e\" (UID: \"ce1e4866-6131-4258-a046-06f123decc1e\") " Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578425 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.578730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578686 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.579172 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.578917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.579340 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.579315 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.579754 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.579727 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.580589 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.580553 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:55.580700 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.580593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.580841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.580815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out" (OuterVolumeSpecName: "config-out") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:55.580920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.580823 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.580976 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.580934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.581285 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.581174 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.581376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.581319 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.582212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.581953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.582212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.582180 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:55.582526 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.582500 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:55.582850 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.582822 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.583287 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.583266 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt" (OuterVolumeSpecName: "kube-api-access-jmslt") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "kube-api-access-jmslt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:55.583431 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.583415 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config" (OuterVolumeSpecName: "config") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.583505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.583434 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.583594 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.583578 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.592094 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.592074 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config" (OuterVolumeSpecName: "web-config") pod "ce1e4866-6131-4258-a046-06f123decc1e" (UID: "ce1e4866-6131-4258-a046-06f123decc1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:55.679632 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679592 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-web-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679632 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679630 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679645 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679658 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679670 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679682 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-tls-assets\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679693 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-config-out\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679703 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-kube-rbac-proxy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679715 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-db\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679728 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679740 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679751 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmslt\" (UniqueName: \"kubernetes.io/projected/ce1e4866-6131-4258-a046-06f123decc1e-kube-api-access-jmslt\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679763 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-grpc-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679776 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679791 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-metrics-client-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679806 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.679816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679819 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce1e4866-6131-4258-a046-06f123decc1e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.680300 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679835 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-metrics-client-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.680300 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.679848 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce1e4866-6131-4258-a046-06f123decc1e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:31:55.822757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822726 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" exitCode=0 Sep 17 18:31:55.822757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822748 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" exitCode=0 Sep 17 18:31:55.822757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822755 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" exitCode=0 Sep 17 18:31:55.822757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822760 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" exitCode=0 Sep 17 18:31:55.822757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822765 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" exitCode=0 Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822770 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce1e4866-6131-4258-a046-06f123decc1e" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" exitCode=0 Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822805 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822807 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822942 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822957 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.822982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} Sep 17 18:31:55.823015 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.823006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce1e4866-6131-4258-a046-06f123decc1e","Type":"ContainerDied","Data":"5ca12c78f8c09a671f29e6e954f3b77883011d201a5a5c774f9edaf455edb6ac"} Sep 17 18:31:55.830834 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.830816 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.838332 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.838317 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.844762 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.844744 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.849158 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.849138 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:55.851798 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.851731 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.853906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.853888 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:55.858690 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.858677 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.865511 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.865491 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.871890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.871873 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.872140 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.872123 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.872188 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872148 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.872188 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872170 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.872424 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.872407 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.872478 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872432 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.872478 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872475 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.872716 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.872698 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.872770 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872722 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.872770 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.872744 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.872997 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.872981 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.873044 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873002 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.873044 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873015 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.873250 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.873231 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.873322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873256 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.873322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873279 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.873551 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.873533 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.873597 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873555 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.873597 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873567 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.873830 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:31:55.873803 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.873890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873826 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.873890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.873840 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.874115 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874097 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.874163 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874125 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.874357 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874341 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.874401 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874358 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.874595 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874578 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.874595 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874594 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.874847 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874833 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.874884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.874848 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.875066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875051 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.875123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875066 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.875265 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875248 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.875305 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875265 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.875501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875442 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.875501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875504 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.875742 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875725 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.875797 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875743 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.875935 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875919 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.875975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.875935 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.876121 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876106 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.876160 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876121 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.876313 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876297 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.876354 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876314 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.876514 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876493 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.876559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876516 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.876742 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876719 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.876853 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876743 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.877001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.876986 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.877047 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877001 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.877203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877183 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.877271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877204 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.877413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877398 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.877475 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877413 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.877622 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877608 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.877666 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877622 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.877806 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877790 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.877869 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.877807 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.878062 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878036 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.878062 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878060 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.878323 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878302 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.878323 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878319 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.878669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878643 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.878669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878664 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.878934 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878917 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.878934 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.878933 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.879169 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879145 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.879248 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879179 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.879442 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879422 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.879442 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879442 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.879778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879748 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.879778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.879772 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.880034 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880013 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.880108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880035 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.880283 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880257 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.880361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880288 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.880361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880338 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:55.880571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880553 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.880621 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880572 2559 scope.go:117] "RemoveContainer" containerID="874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f" Sep 17 18:31:55.880662 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880647 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="config-reloader" Sep 17 18:31:55.880662 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880660 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="config-reloader" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880674 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerName="registry" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880682 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerName="registry" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880693 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="prometheus" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880702 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="prometheus" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880713 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="init-config-reloader" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880718 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="init-config-reloader" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880727 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880732 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880744 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-thanos" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880752 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-thanos" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880749 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f"} err="failed to get container status \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": rpc error: code = NotFound desc = could not find container \"874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f\": container with ID starting with 874c5258bf33ebb7a7c3cad0ac3938ad3e18d40121d87243931d62ef9cf70c3f not found: ID does not exist" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880759 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-web" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880762 2559 scope.go:117] "RemoveContainer" containerID="2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880764 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-web" Sep 17 18:31:55.880773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880776 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="thanos-sidecar" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880782 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="thanos-sidecar" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880835 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b81a194-047c-45cb-a2ae-4d06c089c2f9" containerName="registry" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880841 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="thanos-sidecar" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880847 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880853 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="prometheus" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880861 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-thanos" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880870 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="kube-rbac-proxy-web" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.880875 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1e4866-6131-4258-a046-06f123decc1e" containerName="config-reloader" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881030 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649"} err="failed to get container status \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": rpc error: code = NotFound desc = could not find container \"2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649\": container with ID starting with 2d35d17a715f2f7fb689cf3e70d7e5b0b3dfe104c2a201b3066d1af0a3af7649 not found: ID does not exist" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881048 2559 scope.go:117] "RemoveContainer" containerID="4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881284 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb"} err="failed to get container status \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": rpc error: code = NotFound desc = could not find container \"4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb\": container with ID starting with 4dc754cd1afe9558e45c73825b5811a62d0b150d908ebf672f0f5c1e9e0754eb not found: ID does not exist" Sep 17 18:31:55.881319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881304 2559 scope.go:117] "RemoveContainer" containerID="4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5" Sep 17 18:31:55.881748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881572 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5"} err="failed to get container status \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": rpc error: code = NotFound desc = could not find container \"4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5\": container with ID starting with 4e9f235e62663f0290fc510dcf2aa4c16a824c9bdbf7c5997ba2a28cb8664cb5 not found: ID does not exist" Sep 17 18:31:55.881748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881589 2559 scope.go:117] "RemoveContainer" containerID="f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612" Sep 17 18:31:55.881841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881808 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612"} err="failed to get container status \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": rpc error: code = NotFound desc = could not find container \"f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612\": container with ID starting with f761a43b5ff754b85c50a1da43011f1bae690fa6aa6556242c3b86b261b4a612 not found: ID does not exist" Sep 17 18:31:55.881841 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.881833 2559 scope.go:117] "RemoveContainer" containerID="5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058" Sep 17 18:31:55.882087 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.882071 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058"} err="failed to get container status \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": rpc error: code = NotFound desc = could not find container \"5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058\": container with ID starting with 5e852759a1ce8062dcd9e337fca665c2eae55d6afd2a438428bfa3f41595e058 not found: ID does not exist" Sep 17 18:31:55.882151 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.882089 2559 scope.go:117] "RemoveContainer" containerID="c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791" Sep 17 18:31:55.882330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.882314 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791"} err="failed to get container status \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": rpc error: code = NotFound desc = could not find container \"c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791\": container with ID starting with c6e50addcc1895215decf934e6bf85f77fc85e2c84fe9eee23107128f256c791 not found: ID does not exist" Sep 17 18:31:55.884376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.884361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.886871 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.886854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:31:55.887119 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:31:55.887251 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887119 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:31:55.887251 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887196 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a74sm9j9avt6b\"" Sep 17 18:31:55.887359 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:31:55.887614 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887596 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:31:55.887827 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887811 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:31:55.887922 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.887898 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:31:55.888046 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888030 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bv2gf\"" Sep 17 18:31:55.888100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888082 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:31:55.888139 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888123 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:31:55.888231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888215 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:31:55.888317 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888297 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:31:55.888317 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:31:55.888773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.888752 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 18:31:55.891228 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.891209 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:31:55.895195 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.895177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:31:55.906847 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.904704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:55.981823 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981795 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.981956 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.981956 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.981956 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c777x\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-kube-api-access-c777x\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.981956 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.981995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.982361 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.982284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:55.992314 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:55.992291 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1e4866-6131-4258-a046-06f123decc1e" path="/var/lib/kubelet/pods/ce1e4866-6131-4258-a046-06f123decc1e/volumes" Sep 17 18:31:56.083467 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083467 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c777x\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-kube-api-access-c777x\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.083920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.084477 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.083970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.084477 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.084312 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.084589 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.084572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.086190 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.085877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.086526 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.086373 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.086696 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.086669 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087325 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087465 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087545 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.087948 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.087915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.088360 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.088247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.088499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.088473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-config\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.088580 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.088521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/524c6f38-1f7f-41a7-82fc-621174e93c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.088758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.088738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.089647 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.089622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/524c6f38-1f7f-41a7-82fc-621174e93c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.089647 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.089634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/524c6f38-1f7f-41a7-82fc-621174e93c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.089769 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.089681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.092191 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.092171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c777x\" (UniqueName: \"kubernetes.io/projected/524c6f38-1f7f-41a7-82fc-621174e93c6f-kube-api-access-c777x\") pod \"prometheus-k8s-0\" (UID: \"524c6f38-1f7f-41a7-82fc-621174e93c6f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.194147 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.194121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:56.470971 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.470939 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:31:56.476440 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:31:56.476369 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524c6f38_1f7f_41a7_82fc_621174e93c6f.slice/crio-a27ced441686c62d68ec6527a9794cd5e73a40948615761490cc7c666239854b WatchSource:0}: Error finding container a27ced441686c62d68ec6527a9794cd5e73a40948615761490cc7c666239854b: Status 404 returned error can't find the container with id a27ced441686c62d68ec6527a9794cd5e73a40948615761490cc7c666239854b Sep 17 18:31:56.828783 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.828708 2559 generic.go:356] "Generic (PLEG): container finished" podID="524c6f38-1f7f-41a7-82fc-621174e93c6f" containerID="2284a3af1f2a945e4834649cde186691f9c5ad1396234e42e323c003ae37be39" exitCode=0 Sep 17 18:31:56.828890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.828798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerDied","Data":"2284a3af1f2a945e4834649cde186691f9c5ad1396234e42e323c003ae37be39"} Sep 17 18:31:56.828890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:56.828872 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"a27ced441686c62d68ec6527a9794cd5e73a40948615761490cc7c666239854b"} Sep 17 18:31:57.837263 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837224 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"9fa02d6fa1da0c9062a0384b17a58dd97776a8e3ba6f9d6d9c548d5e0475f71f"} Sep 17 18:31:57.837609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"98425b031252ad5ed6146a872d29cfe81e8054e6f43a0d2c5f684ce6f9980e3e"} Sep 17 18:31:57.837609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"afac5f74d91b2730948d671ef847bbbc29786182705087c2feded88d36002998"} Sep 17 18:31:57.837609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"9485b6499bd75cfe0c26d13a6b4b8f4e4536dbfe64bb366dfa08dd79d9cc26d3"} Sep 17 18:31:57.837609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837434 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"95f0d4a3451e099cc89d01f05fac3cc7d7e2c6eba3c15219176079e1f02f39d4"} Sep 17 18:31:57.837609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.837492 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"524c6f38-1f7f-41a7-82fc-621174e93c6f","Type":"ContainerStarted","Data":"96e6bad24650c3bd365066ecaf3ecf199c0a529ee276eb22acba146218326ce8"} Sep 17 18:31:57.868106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:31:57.868060 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.849513138 podStartE2EDuration="2.868043463s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.530325ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:57.865362493 +0000 UTC m=+169.853978745" watchObservedRunningTime="2026-09-17 18:31:57.868043463 +0000 UTC m=+169.856659773" Sep 17 18:32:01.194543 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:32:01.194494 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:56.194335 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:32:56.194298 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:56.209180 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:32:56.209155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:57.079344 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:32:57.079317 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:34:09.857990 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:34:09.857956 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:34:09.865809 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:34:09.865787 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:34:09.866430 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:34:09.866413 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:34:09.869958 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:34:09.869941 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 18:34:09.873637 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:34:09.873620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:36:33.205057 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.205021 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6z876"] Sep 17 18:36:33.207673 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.207652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6z876" Sep 17 18:36:33.211706 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.211675 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:36:33.211846 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.211713 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 17 18:36:33.212368 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.212350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 17 18:36:33.212806 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.212773 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:36:33.212934 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.212821 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\"" Sep 17 18:36:33.220843 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.220817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6z876"] Sep 17 18:36:33.286357 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.286334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s69h\" (UniqueName: \"kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.286489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.286376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.387571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.387546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.387664 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.387602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s69h\" (UniqueName: \"kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.387904 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.387885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.397676 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.397655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s69h\" (UniqueName: \"kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h\") pod \"s3-init-6z876\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " pod="kserve/s3-init-6z876" Sep 17 18:36:33.529926 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.529870 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6z876" Sep 17 18:36:33.810128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.810101 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6z876"] Sep 17 18:36:33.811894 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:36:33.811867 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a6ade4b_fc47_49d6_a4dc_9b113eef1d81.slice/crio-7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851 WatchSource:0}: Error finding container 7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851: Status 404 returned error can't find the container with id 7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851 Sep 17 18:36:33.814043 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.814023 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:36:33.905463 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:33.905419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerStarted","Data":"7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851"} Sep 17 18:36:37.926332 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:37.926290 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerStarted","Data":"070a0c587ad43c91d497b3aba3211c1fe0e8bf0bcd99c3bdc54accecb5d844a5"} Sep 17 18:36:39.937953 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:39.937912 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerID="070a0c587ad43c91d497b3aba3211c1fe0e8bf0bcd99c3bdc54accecb5d844a5" exitCode=0 Sep 17 18:36:39.938348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:39.937968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerDied","Data":"070a0c587ad43c91d497b3aba3211c1fe0e8bf0bcd99c3bdc54accecb5d844a5"} Sep 17 18:36:44.961531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:44.961502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerStarted","Data":"51919d9827491e05f40ce8745c6ce5d193692efa18ff1e40f60a7be7835f87be"} Sep 17 18:36:44.990614 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:36:44.990568 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6z876" podStartSLOduration=2.044941908 podStartE2EDuration="11.990553787s" totalImagesPullingTime="8.207850744s" totalInitContainerRuntime="1.737761135s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:44.988623533 +0000 UTC m=+456.977239781" watchObservedRunningTime="2026-09-17 18:36:44.990553787 +0000 UTC m=+456.979170026" Sep 17 18:37:50.211013 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:50.210976 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerID="51919d9827491e05f40ce8745c6ce5d193692efa18ff1e40f60a7be7835f87be" exitCode=0 Sep 17 18:37:50.211013 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:50.211017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerDied","Data":"51919d9827491e05f40ce8745c6ce5d193692efa18ff1e40f60a7be7835f87be"} Sep 17 18:37:51.342303 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.342282 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6z876" Sep 17 18:37:51.469863 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.469799 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s69h\" (UniqueName: \"kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h\") pod \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " Sep 17 18:37:51.469863 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.469850 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache\") pod \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\" (UID: \"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81\") " Sep 17 18:37:51.471915 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.471889 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h" (OuterVolumeSpecName: "kube-api-access-8s69h") pod "6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" (UID: "6a6ade4b-fc47-49d6-a4dc-9b113eef1d81"). InnerVolumeSpecName "kube-api-access-8s69h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:51.492056 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.492021 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" (UID: "6a6ade4b-fc47-49d6-a4dc-9b113eef1d81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:37:51.571044 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.571009 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s69h\" (UniqueName: \"kubernetes.io/projected/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-kube-api-access-8s69h\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.571044 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:51.571044 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6ade4b-fc47-49d6-a4dc-9b113eef1d81-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:37:52.219869 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:52.219835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6z876" event={"ID":"6a6ade4b-fc47-49d6-a4dc-9b113eef1d81","Type":"ContainerDied","Data":"7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851"} Sep 17 18:37:52.219869 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:52.219875 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e77b2b9bfa69eefbeed6a560141d4047c03c9ca67bbd98a4f2c32d09fa3b851" Sep 17 18:37:52.220069 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:37:52.219872 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6z876" Sep 17 18:38:02.190604 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.190567 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6svb5"] Sep 17 18:38:02.191144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.190985 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerName="download-hf-model" Sep 17 18:38:02.191144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.191003 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerName="download-hf-model" Sep 17 18:38:02.191144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.191016 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerName="s3-init" Sep 17 18:38:02.191144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.191025 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerName="s3-init" Sep 17 18:38:02.191144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.191110 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6ade4b-fc47-49d6-a4dc-9b113eef1d81" containerName="s3-init" Sep 17 18:38:02.194197 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.194175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.196814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.196790 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Sep 17 18:38:02.197816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.197798 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 17 18:38:02.197889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.197836 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:38:02.197889 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.197796 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\"" Sep 17 18:38:02.197975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.197913 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:38:02.201898 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.201879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6svb5"] Sep 17 18:38:02.355078 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.355050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhdb\" (UniqueName: \"kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.355206 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.355089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.456403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.456336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhdb\" (UniqueName: \"kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.456403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.456379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.456782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.456766 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.465123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.465095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhdb\" (UniqueName: \"kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb\") pod \"s3-tls-init-custom-6svb5\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.509520 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.509499 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:02.776628 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:02.776607 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6svb5"] Sep 17 18:38:02.779113 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:38:02.779085 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459fb52c_abc0_41db_b840_a0bd71c48c7a.slice/crio-79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e WatchSource:0}: Error finding container 79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e: Status 404 returned error can't find the container with id 79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e Sep 17 18:38:03.266879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:03.266844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerStarted","Data":"079515a76b95382acaa80be65ac36e40b10a176d5eb48cbf253bf99fea7511dd"} Sep 17 18:38:03.266879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:03.266883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerStarted","Data":"79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e"} Sep 17 18:38:05.276233 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:05.276199 2559 generic.go:356] "Generic (PLEG): container finished" podID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerID="079515a76b95382acaa80be65ac36e40b10a176d5eb48cbf253bf99fea7511dd" exitCode=0 Sep 17 18:38:05.276617 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:05.276269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerDied","Data":"079515a76b95382acaa80be65ac36e40b10a176d5eb48cbf253bf99fea7511dd"} Sep 17 18:38:06.282131 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:06.282101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerStarted","Data":"4c71b812d6db8b774dde42fe12af28d83cf843c6af9ebe752c02338c10c89020"} Sep 17 18:38:06.300113 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:06.300071 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6svb5" podStartSLOduration=2.649361448 podStartE2EDuration="4.300056437s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.650694989s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:06.29878051 +0000 UTC m=+538.287396758" watchObservedRunningTime="2026-09-17 18:38:06.300056437 +0000 UTC m=+538.288672664" Sep 17 18:38:12.310050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:12.309968 2559 generic.go:356] "Generic (PLEG): container finished" podID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerID="4c71b812d6db8b774dde42fe12af28d83cf843c6af9ebe752c02338c10c89020" exitCode=0 Sep 17 18:38:12.310050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:12.310038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerDied","Data":"4c71b812d6db8b774dde42fe12af28d83cf843c6af9ebe752c02338c10c89020"} Sep 17 18:38:13.446809 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.446789 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:13.543785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.543759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache\") pod \"459fb52c-abc0-41db-b840-a0bd71c48c7a\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " Sep 17 18:38:13.543881 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.543809 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qhdb\" (UniqueName: \"kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb\") pod \"459fb52c-abc0-41db-b840-a0bd71c48c7a\" (UID: \"459fb52c-abc0-41db-b840-a0bd71c48c7a\") " Sep 17 18:38:13.545777 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.545752 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb" (OuterVolumeSpecName: "kube-api-access-8qhdb") pod "459fb52c-abc0-41db-b840-a0bd71c48c7a" (UID: "459fb52c-abc0-41db-b840-a0bd71c48c7a"). InnerVolumeSpecName "kube-api-access-8qhdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:13.566086 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.566036 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "459fb52c-abc0-41db-b840-a0bd71c48c7a" (UID: "459fb52c-abc0-41db-b840-a0bd71c48c7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:13.645116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.645086 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qhdb\" (UniqueName: \"kubernetes.io/projected/459fb52c-abc0-41db-b840-a0bd71c48c7a-kube-api-access-8qhdb\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:38:13.645116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:13.645112 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459fb52c-abc0-41db-b840-a0bd71c48c7a-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:38:14.319887 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:14.319814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6svb5" event={"ID":"459fb52c-abc0-41db-b840-a0bd71c48c7a","Type":"ContainerDied","Data":"79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e"} Sep 17 18:38:14.319887 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:14.319848 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ef1da8065cd90f92d7d5aa64ebcd9826b2929b10d13767a28648faf2b15a9e" Sep 17 18:38:14.319887 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:14.319855 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6svb5" Sep 17 18:38:17.449747 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.449715 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-c2rkn"] Sep 17 18:38:17.450104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.450011 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerName="s3-tls-init-custom" Sep 17 18:38:17.450104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.450022 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerName="s3-tls-init-custom" Sep 17 18:38:17.450104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.450036 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerName="download-hf-model" Sep 17 18:38:17.450104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.450041 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerName="download-hf-model" Sep 17 18:38:17.450104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.450091 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="459fb52c-abc0-41db-b840-a0bd71c48c7a" containerName="s3-tls-init-custom" Sep 17 18:38:17.452214 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.452196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.455755 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.455735 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 17 18:38:17.456081 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.456064 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\"" Sep 17 18:38:17.457322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.457306 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:38:17.457393 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.457364 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Sep 17 18:38:17.457898 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.457881 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:38:17.463607 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.463584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-c2rkn"] Sep 17 18:38:17.474018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.473996 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.474124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.474108 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7pb\" (UniqueName: \"kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.575050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.575016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7pb\" (UniqueName: \"kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.575204 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.575055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.575499 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.575479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.583374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.583355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7pb\" (UniqueName: \"kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb\") pod \"s3-tls-init-serving-c2rkn\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:17.780608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:17.780548 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:18.047352 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:18.047309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-c2rkn"] Sep 17 18:38:18.049622 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:38:18.049589 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6e017e_fb4e_4d60_aff2_cd42d7df536f.slice/crio-f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc WatchSource:0}: Error finding container f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc: Status 404 returned error can't find the container with id f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc Sep 17 18:38:18.337221 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:18.337133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerStarted","Data":"05f5ffd8b9492778e7efa283dc43e76c94ec1c4bc9ad233563414248770f7d93"} Sep 17 18:38:18.337221 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:18.337179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerStarted","Data":"f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc"} Sep 17 18:38:20.349204 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:20.349064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerDied","Data":"05f5ffd8b9492778e7efa283dc43e76c94ec1c4bc9ad233563414248770f7d93"} Sep 17 18:38:20.349204 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:20.348946 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerID="05f5ffd8b9492778e7efa283dc43e76c94ec1c4bc9ad233563414248770f7d93" exitCode=0 Sep 17 18:38:21.355113 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:21.355079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerStarted","Data":"34905e067b1bdf22733ca0677555ffe209ceb7fe4be7d61686f6218871255f33"} Sep 17 18:38:21.372808 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:21.372762 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-c2rkn" podStartSLOduration=2.539000843 podStartE2EDuration="4.372746744s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.833745901s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:21.371583382 +0000 UTC m=+553.360199633" watchObservedRunningTime="2026-09-17 18:38:21.372746744 +0000 UTC m=+553.361362993" Sep 17 18:38:27.380106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:27.380013 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerID="34905e067b1bdf22733ca0677555ffe209ceb7fe4be7d61686f6218871255f33" exitCode=0 Sep 17 18:38:27.380106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:27.380084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerDied","Data":"34905e067b1bdf22733ca0677555ffe209ceb7fe4be7d61686f6218871255f33"} Sep 17 18:38:28.515371 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.515350 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:28.563906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.563878 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht7pb\" (UniqueName: \"kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb\") pod \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " Sep 17 18:38:28.564050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.563928 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache\") pod \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\" (UID: \"7a6e017e-fb4e-4d60-aff2-cd42d7df536f\") " Sep 17 18:38:28.565872 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.565842 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb" (OuterVolumeSpecName: "kube-api-access-ht7pb") pod "7a6e017e-fb4e-4d60-aff2-cd42d7df536f" (UID: "7a6e017e-fb4e-4d60-aff2-cd42d7df536f"). InnerVolumeSpecName "kube-api-access-ht7pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:28.587051 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.587027 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a6e017e-fb4e-4d60-aff2-cd42d7df536f" (UID: "7a6e017e-fb4e-4d60-aff2-cd42d7df536f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:38:28.665403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.665374 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht7pb\" (UniqueName: \"kubernetes.io/projected/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-kube-api-access-ht7pb\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:38:28.665403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:28.665399 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e017e-fb4e-4d60-aff2-cd42d7df536f-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:38:29.389522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:29.389497 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2rkn" Sep 17 18:38:29.389668 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:29.389502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2rkn" event={"ID":"7a6e017e-fb4e-4d60-aff2-cd42d7df536f","Type":"ContainerDied","Data":"f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc"} Sep 17 18:38:29.389668 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:29.389595 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f49b0dc56f219e7fdc0ccd3c2fa56ca4499dfff455f3b1d40b1857ee9d9f1bfc" Sep 17 18:38:41.226780 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.226741 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:38:41.227207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.227028 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerName="s3-tls-init-serving" Sep 17 18:38:41.227207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.227039 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerName="s3-tls-init-serving" Sep 17 18:38:41.227207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.227049 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerName="download-hf-model" Sep 17 18:38:41.227207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.227055 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerName="download-hf-model" Sep 17 18:38:41.227207 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.227112 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6e017e-fb4e-4d60-aff2-cd42d7df536f" containerName="s3-tls-init-serving" Sep 17 18:38:41.229824 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.229809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.233292 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.233271 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:38:41.233464 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.233437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\"" Sep 17 18:38:41.233588 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.233569 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\"" Sep 17 18:38:41.234558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.234526 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 18:38:41.234558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.234532 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:38:41.234708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.234621 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:38:41.234708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.234665 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:38:41.246440 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.246421 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:38:41.358626 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.358603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.358743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.358637 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.358743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.358656 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.358743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.358675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.358743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.358699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wm9f\" (UniqueName: \"kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459189 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459298 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459298 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459298 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wm9f\" (UniqueName: \"kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459298 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459576 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459929 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.459973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.459914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.461548 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.461532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.467406 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.467385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wm9f\" (UniqueName: \"kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f\") pod \"isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.539814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.539755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:38:41.809870 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:41.809843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:38:41.811597 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:38:41.811568 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e70f071_0f60_4b9f_8ced_1503a1ac8ebb.slice/crio-c8b6d5dd30687ea447a6a7a3cd9655f038b7b79992abf876f2132389cde9e701 WatchSource:0}: Error finding container c8b6d5dd30687ea447a6a7a3cd9655f038b7b79992abf876f2132389cde9e701: Status 404 returned error can't find the container with id c8b6d5dd30687ea447a6a7a3cd9655f038b7b79992abf876f2132389cde9e701 Sep 17 18:38:42.442487 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:42.442431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerStarted","Data":"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f"} Sep 17 18:38:42.442870 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:42.442539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerStarted","Data":"c8b6d5dd30687ea447a6a7a3cd9655f038b7b79992abf876f2132389cde9e701"} Sep 17 18:38:46.460181 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:46.460141 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerID="154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f" exitCode=0 Sep 17 18:38:46.460607 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:38:46.460214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerDied","Data":"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f"} Sep 17 18:39:01.541178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:01.541139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerStarted","Data":"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f"} Sep 17 18:39:03.553399 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:03.553359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerStarted","Data":"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525"} Sep 17 18:39:06.570398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:06.570351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerStarted","Data":"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af"} Sep 17 18:39:06.570859 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:06.570633 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:39:06.594647 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:06.594602 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podStartSLOduration=2.790978533 podStartE2EDuration="25.594587008s" totalImagesPullingTime="19.028507804s" totalInitContainerRuntime="3.775100671s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.593068679 +0000 UTC m=+598.581684941" watchObservedRunningTime="2026-09-17 18:39:06.594587008 +0000 UTC m=+598.583203257" Sep 17 18:39:07.574796 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:07.574758 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:39:07.575172 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:07.574832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:39:07.576056 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:07.576024 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:07.576747 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:07.576721 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:08.595895 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:08.595853 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:08.597958 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:08.597924 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:08.608700 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:08.608681 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:39:09.599804 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.599763 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:09.600195 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.600135 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:09.889459 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.889423 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:39:09.893072 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.893053 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:39:09.898025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.898003 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:39:09.902092 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:09.902071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:39:19.600083 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:19.600032 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:19.600579 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:19.600499 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:29.600177 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:29.600136 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:29.600634 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:29.600611 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:39.599881 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:39.599836 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:39.600310 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:39.600269 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:49.600084 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:49.600038 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:49.600594 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:49.600567 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:39:59.600711 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:59.600663 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:39:59.601293 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:39:59.601226 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:09.600321 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:09.600276 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:40:09.600856 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:09.600830 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:11.992376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:11.992348 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:11.992761 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:11.992417 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:26.254719 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.254684 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:40:26.255227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.254953 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" containerID="cri-o://c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f" gracePeriod=30 Sep 17 18:40:26.255227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.255016 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" containerID="cri-o://93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525" gracePeriod=30 Sep 17 18:40:26.255227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.254997 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" containerID="cri-o://cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af" gracePeriod=30 Sep 17 18:40:26.454751 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.454714 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:40:26.457394 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.457378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.460138 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.460112 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\"" Sep 17 18:40:26.460262 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.460119 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\"" Sep 17 18:40:26.470937 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.470913 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:40:26.520508 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.520417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.520646 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.520520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.520646 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.520567 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tghj\" (UniqueName: \"kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.520646 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.520607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.520749 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.520678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.621398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.621361 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tghj\" (UniqueName: \"kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.621610 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.621412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.621610 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:26.621537 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found Sep 17 18:40:26.621610 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:26.621590 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls podName:dce1805d-8afe-48d9-bb60-c68da19edbca nodeName:}" failed. No retries permitted until 2026-09-17 18:40:27.121571022 +0000 UTC m=+679.110187248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found Sep 17 18:40:26.621748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.621606 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.621748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.621652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.621748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.621700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.622085 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.622055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.622291 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.622272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.622328 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.622318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.631227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.631198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tghj\" (UniqueName: \"kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:26.919149 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.919116 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerID="93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525" exitCode=2 Sep 17 18:40:26.919315 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:26.919177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerDied","Data":"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525"} Sep 17 18:40:27.126355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.126319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:27.128884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.128860 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:27.367181 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.367089 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:27.636585 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.636521 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:40:27.638980 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:40:27.638948 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce1805d_8afe_48d9_bb60_c68da19edbca.slice/crio-37fd7bacd5be6c0efa6e4b75a43d1d971632cb8dba89adb78177c2affa06eff2 WatchSource:0}: Error finding container 37fd7bacd5be6c0efa6e4b75a43d1d971632cb8dba89adb78177c2affa06eff2: Status 404 returned error can't find the container with id 37fd7bacd5be6c0efa6e4b75a43d1d971632cb8dba89adb78177c2affa06eff2 Sep 17 18:40:27.926217 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.926177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerStarted","Data":"8b3caae4a81b8bd8c41c655281da50fd6c3dd27eda2693c0438376190d98afdf"} Sep 17 18:40:27.926382 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:27.926256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerStarted","Data":"37fd7bacd5be6c0efa6e4b75a43d1d971632cb8dba89adb78177c2affa06eff2"} Sep 17 18:40:28.596559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:28.596519 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:30.940665 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:30.940579 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerID="c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f" exitCode=0 Sep 17 18:40:30.941003 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:30.940660 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerDied","Data":"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f"} Sep 17 18:40:31.946759 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:31.946722 2559 generic.go:356] "Generic (PLEG): container finished" podID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerID="8b3caae4a81b8bd8c41c655281da50fd6c3dd27eda2693c0438376190d98afdf" exitCode=0 Sep 17 18:40:31.947271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:31.946793 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerDied","Data":"8b3caae4a81b8bd8c41c655281da50fd6c3dd27eda2693c0438376190d98afdf"} Sep 17 18:40:31.987920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:31.987878 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:40:31.988348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:31.988314 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:32.954293 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.954252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerStarted","Data":"abe9275ab8b524f09f5e24203ace79e19b5fd1897a0466070bd2ba35eeea81fd"} Sep 17 18:40:32.954800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.954353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerStarted","Data":"4f625421c218f3dd416c01a0ecf59e0731b52fd2f8dff4fef3525d2b8745d290"} Sep 17 18:40:32.954800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.954381 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerStarted","Data":"5fd5ce5ea0bec9ef2733f540cdc59f1906aec0e69099784514cf0e0c81226b8c"} Sep 17 18:40:32.954800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.954620 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:32.954800 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.954785 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:32.955930 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.955896 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:32.977834 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:32.977790 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podStartSLOduration=3.405221768 podStartE2EDuration="6.977777914s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.572556146s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:32.976149903 +0000 UTC m=+684.964766153" watchObservedRunningTime="2026-09-17 18:40:32.977777914 +0000 UTC m=+684.966394163" Sep 17 18:40:33.596864 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:33.596827 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:33.959814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:33.959764 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:33.960273 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:33.959872 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:33.960977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:33.960949 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:34.964380 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:34.964346 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:34.964829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:34.964723 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:38.596972 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:38.596932 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:38.597372 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:38.597045 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:39.968914 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:39.968880 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:40:39.969543 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:39.969505 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:39.969900 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:39.969877 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:41.988815 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:41.988767 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:40:41.989250 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:41.989126 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:43.597009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:43.596962 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:48.597235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:48.597192 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:49.970087 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:49.970044 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:49.970609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:49.970586 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:51.987875 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:51.987777 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 18:40:51.988294 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:51.988128 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:40:51.992401 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:51.992377 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:51.992527 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:51.992507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:53.597163 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:53.597120 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 18:40:56.398492 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.398467 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:56.542239 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls\") pod \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " Sep 17 18:40:56.542239 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wm9f\" (UniqueName: \"kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f\") pod \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " Sep 17 18:40:56.542431 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542260 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert\") pod \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " Sep 17 18:40:56.542431 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542299 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location\") pod \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " Sep 17 18:40:56.542431 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config\") pod \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\" (UID: \"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb\") " Sep 17 18:40:56.542650 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542616 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" (UID: "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:40:56.542708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542681 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" (UID: "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:40:56.542771 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.542739 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" (UID: "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:40:56.544402 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.544381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" (UID: "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:40:56.544555 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.544508 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f" (OuterVolumeSpecName: "kube-api-access-5wm9f") pod "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" (UID: "8e70f071-0f60-4b9f-8ced-1503a1ac8ebb"). InnerVolumeSpecName "kube-api-access-5wm9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:40:56.643584 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.643562 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wm9f\" (UniqueName: \"kubernetes.io/projected/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-api-access-5wm9f\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:40:56.643584 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.643584 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:40:56.643718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.643593 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:40:56.643718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.643602 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:40:56.643718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:56.643611 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:40:57.058654 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.058628 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerID="cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af" exitCode=0 Sep 17 18:40:57.058784 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.058685 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" Sep 17 18:40:57.058784 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.058709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerDied","Data":"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af"} Sep 17 18:40:57.058784 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.058775 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv" event={"ID":"8e70f071-0f60-4b9f-8ced-1503a1ac8ebb","Type":"ContainerDied","Data":"c8b6d5dd30687ea447a6a7a3cd9655f038b7b79992abf876f2132389cde9e701"} Sep 17 18:40:57.058879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.058811 2559 scope.go:117] "RemoveContainer" containerID="cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af" Sep 17 18:40:57.068944 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.068805 2559 scope.go:117] "RemoveContainer" containerID="93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525" Sep 17 18:40:57.077643 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.077597 2559 scope.go:117] "RemoveContainer" containerID="c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f" Sep 17 18:40:57.085842 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.085815 2559 scope.go:117] "RemoveContainer" containerID="154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f" Sep 17 18:40:57.086112 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.086092 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:40:57.090198 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.090175 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-564ccdbdcc-9w8vv"] Sep 17 18:40:57.093474 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.093442 2559 scope.go:117] "RemoveContainer" containerID="cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af" Sep 17 18:40:57.093762 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:57.093740 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af\": container with ID starting with cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af not found: ID does not exist" containerID="cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af" Sep 17 18:40:57.093849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.093768 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af"} err="failed to get container status \"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af\": rpc error: code = NotFound desc = could not find container \"cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af\": container with ID starting with cb46f818898e39113be1e28d5d70086dd0db83a5ea779d349cfe622369a434af not found: ID does not exist" Sep 17 18:40:57.093849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.093785 2559 scope.go:117] "RemoveContainer" containerID="93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525" Sep 17 18:40:57.094044 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:57.094028 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525\": container with ID starting with 93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525 not found: ID does not exist" containerID="93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525" Sep 17 18:40:57.094087 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.094048 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525"} err="failed to get container status \"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525\": rpc error: code = NotFound desc = could not find container \"93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525\": container with ID starting with 93f5ba415498758aaa314f21bb181eb4524fb8362d2c829054f6b0a55bb35525 not found: ID does not exist" Sep 17 18:40:57.094087 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.094063 2559 scope.go:117] "RemoveContainer" containerID="c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f" Sep 17 18:40:57.094308 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:57.094291 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f\": container with ID starting with c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f not found: ID does not exist" containerID="c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f" Sep 17 18:40:57.094360 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.094310 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f"} err="failed to get container status \"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f\": rpc error: code = NotFound desc = could not find container \"c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f\": container with ID starting with c9c14a95c463bae71c76cbd9189268de1420527c643e33711ea2eff2fc627b9f not found: ID does not exist" Sep 17 18:40:57.094360 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.094322 2559 scope.go:117] "RemoveContainer" containerID="154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f" Sep 17 18:40:57.094552 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:40:57.094531 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f\": container with ID starting with 154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f not found: ID does not exist" containerID="154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f" Sep 17 18:40:57.094629 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.094556 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f"} err="failed to get container status \"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f\": rpc error: code = NotFound desc = could not find container \"154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f\": container with ID starting with 154b2209d68ffdab721713f2cf947df8d348d9d756c33bf472793ad5400c408f not found: ID does not exist" Sep 17 18:40:57.992156 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:57.992124 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" path="/var/lib/kubelet/pods/8e70f071-0f60-4b9f-8ced-1503a1ac8ebb/volumes" Sep 17 18:40:59.970202 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:59.970166 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:40:59.970684 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:40:59.970658 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:41:09.969377 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:09.969338 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:41:09.969879 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:09.969852 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:41:19.970074 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:19.970036 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:41:19.970492 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:19.970468 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:41:29.969661 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:29.969613 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:41:29.970144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:29.970117 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:41:39.969696 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:39.969647 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:41:39.970189 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:39.970160 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:41:49.970536 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:49.970510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:41:49.970942 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:41:49.970623 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:01.423642 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.423610 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:42:01.424053 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.423868 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" containerID="cri-o://5fd5ce5ea0bec9ef2733f540cdc59f1906aec0e69099784514cf0e0c81226b8c" gracePeriod=30 Sep 17 18:42:01.424053 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.423908 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" containerID="cri-o://abe9275ab8b524f09f5e24203ace79e19b5fd1897a0466070bd2ba35eeea81fd" gracePeriod=30 Sep 17 18:42:01.424941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.424271 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" containerID="cri-o://4f625421c218f3dd416c01a0ecf59e0731b52fd2f8dff4fef3525d2b8745d290" gracePeriod=30 Sep 17 18:42:01.544773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.544748 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:42:01.545059 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545047 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" Sep 17 18:42:01.545100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545060 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" Sep 17 18:42:01.545100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545079 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" Sep 17 18:42:01.545100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545084 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" Sep 17 18:42:01.545100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545093 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" Sep 17 18:42:01.545100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545098 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" Sep 17 18:42:01.545243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545106 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="storage-initializer" Sep 17 18:42:01.545243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545111 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="storage-initializer" Sep 17 18:42:01.545243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545164 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="agent" Sep 17 18:42:01.545243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545171 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kube-rbac-proxy" Sep 17 18:42:01.545243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.545178 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e70f071-0f60-4b9f-8ced-1503a1ac8ebb" containerName="kserve-container" Sep 17 18:42:01.547938 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.547924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.551056 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.551038 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\"" Sep 17 18:42:01.551570 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.551555 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\"" Sep 17 18:42:01.559011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.558990 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:42:01.616212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.616190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.616343 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.616219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.616343 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.616240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfj8\" (UniqueName: \"kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.717336 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.717280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.717336 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.717309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.717336 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.717332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfj8\" (UniqueName: \"kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.717968 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.717950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.719743 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.719724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.725551 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.725520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfj8\" (UniqueName: \"kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8\") pod \"message-dumper-predictor-6575bcfdfd-6gkjn\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:01.857791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:01.857758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:02.128195 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:02.128146 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:42:02.130271 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:42:02.130238 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aba06a_ba4f_4ff9_9746_ffe3625ec084.slice/crio-f2895eab40a68c47bdff50b24be384c8dc85b2bca9d6062321d1a285cb03f474 WatchSource:0}: Error finding container f2895eab40a68c47bdff50b24be384c8dc85b2bca9d6062321d1a285cb03f474: Status 404 returned error can't find the container with id f2895eab40a68c47bdff50b24be384c8dc85b2bca9d6062321d1a285cb03f474 Sep 17 18:42:02.132016 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:02.132001 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:42:02.331288 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:02.331251 2559 generic.go:356] "Generic (PLEG): container finished" podID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerID="4f625421c218f3dd416c01a0ecf59e0731b52fd2f8dff4fef3525d2b8745d290" exitCode=2 Sep 17 18:42:02.331422 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:02.331326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerDied","Data":"4f625421c218f3dd416c01a0ecf59e0731b52fd2f8dff4fef3525d2b8745d290"} Sep 17 18:42:02.332518 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:02.332493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerStarted","Data":"f2895eab40a68c47bdff50b24be384c8dc85b2bca9d6062321d1a285cb03f474"} Sep 17 18:42:04.343129 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.343095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerStarted","Data":"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3"} Sep 17 18:42:04.343596 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.343152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerStarted","Data":"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5"} Sep 17 18:42:04.343596 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.343177 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:04.343596 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.343218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:04.345056 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.345035 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:04.362101 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.362057 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" podStartSLOduration=2.109429784 podStartE2EDuration="3.362044273s" totalImagesPullingTime="1.252614489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:04.360109756 +0000 UTC m=+776.348726033" watchObservedRunningTime="2026-09-17 18:42:04.362044273 +0000 UTC m=+776.350660521" Sep 17 18:42:04.964660 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:04.964621 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:06.353616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:06.353584 2559 generic.go:356] "Generic (PLEG): container finished" podID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerID="5fd5ce5ea0bec9ef2733f540cdc59f1906aec0e69099784514cf0e0c81226b8c" exitCode=0 Sep 17 18:42:06.353949 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:06.353658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerDied","Data":"5fd5ce5ea0bec9ef2733f540cdc59f1906aec0e69099784514cf0e0c81226b8c"} Sep 17 18:42:09.965046 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:09.965008 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:09.969482 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:09.969438 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:42:09.969621 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:09.969600 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:11.359488 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:11.359459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:42:14.965087 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:14.965047 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:14.965502 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:14.965214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:19.965195 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:19.965158 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:19.969570 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:19.969539 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:42:19.969935 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:19.969910 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:21.602289 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.602214 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:42:21.605682 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.605667 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.608669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.608642 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\"" Sep 17 18:42:21.608669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.608652 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\"" Sep 17 18:42:21.616688 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.616665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:42:21.778554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.778527 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.778554 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.778558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.778721 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.778577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.778721 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.778592 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.778721 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.778705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz58r\" (UniqueName: \"kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879476 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879476 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879476 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879731 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879489 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879731 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz58r\" (UniqueName: \"kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.879900 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.879877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.880180 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.880145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.880231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.880206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.881913 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.881886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.888828 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.888805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz58r\" (UniqueName: \"kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r\") pod \"isvc-logger-predictor-5d79fc8c86-kspg7\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:21.915643 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:21.915612 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:22.189537 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:22.189515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:42:22.191257 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:42:22.191226 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b0163_cdf6_4b33_9761_a8477ba0e39e.slice/crio-d1e1d2c17c5e7f1ed9da8b1a418607b64ddf62ec1ac4360380aff977ebf43594 WatchSource:0}: Error finding container d1e1d2c17c5e7f1ed9da8b1a418607b64ddf62ec1ac4360380aff977ebf43594: Status 404 returned error can't find the container with id d1e1d2c17c5e7f1ed9da8b1a418607b64ddf62ec1ac4360380aff977ebf43594 Sep 17 18:42:22.424508 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:22.424475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerStarted","Data":"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb"} Sep 17 18:42:22.424651 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:22.424539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerStarted","Data":"d1e1d2c17c5e7f1ed9da8b1a418607b64ddf62ec1ac4360380aff977ebf43594"} Sep 17 18:42:24.964877 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:24.964828 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:26.448833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:26.448799 2559 generic.go:356] "Generic (PLEG): container finished" podID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerID="16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb" exitCode=0 Sep 17 18:42:26.449220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:26.448866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerDied","Data":"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb"} Sep 17 18:42:27.456738 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.456699 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerStarted","Data":"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d"} Sep 17 18:42:27.457142 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.456771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerStarted","Data":"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5"} Sep 17 18:42:27.457142 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.456798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerStarted","Data":"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956"} Sep 17 18:42:27.457142 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.457084 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:27.457276 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.457243 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:27.458622 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.458599 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:27.481987 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:27.481944 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podStartSLOduration=2.967632845 podStartE2EDuration="6.481930718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.514297873s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:27.480590469 +0000 UTC m=+799.469206717" watchObservedRunningTime="2026-09-17 18:42:27.481930718 +0000 UTC m=+799.470546966" Sep 17 18:42:28.462189 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:28.462158 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:28.462618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:28.462243 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:28.463255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:28.463233 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:29.466398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.466355 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:29.466887 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.466647 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:29.965106 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.965070 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:42:29.969430 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.969389 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused" Sep 17 18:42:29.969553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.969540 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:29.969696 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.969675 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:29.969789 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:29.969774 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:31.477433 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.477396 2559 generic.go:356] "Generic (PLEG): container finished" podID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerID="abe9275ab8b524f09f5e24203ace79e19b5fd1897a0466070bd2ba35eeea81fd" exitCode=0 Sep 17 18:42:31.477832 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.477472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerDied","Data":"abe9275ab8b524f09f5e24203ace79e19b5fd1897a0466070bd2ba35eeea81fd"} Sep 17 18:42:31.574718 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.574695 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:31.661143 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") pod \"dce1805d-8afe-48d9-bb60-c68da19edbca\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " Sep 17 18:42:31.661268 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661180 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert\") pod \"dce1805d-8afe-48d9-bb60-c68da19edbca\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " Sep 17 18:42:31.661268 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661199 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location\") pod \"dce1805d-8afe-48d9-bb60-c68da19edbca\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " Sep 17 18:42:31.661268 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661240 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tghj\" (UniqueName: \"kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj\") pod \"dce1805d-8afe-48d9-bb60-c68da19edbca\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " Sep 17 18:42:31.661268 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661265 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config\") pod \"dce1805d-8afe-48d9-bb60-c68da19edbca\" (UID: \"dce1805d-8afe-48d9-bb60-c68da19edbca\") " Sep 17 18:42:31.661610 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dce1805d-8afe-48d9-bb60-c68da19edbca" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:42:31.661610 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dce1805d-8afe-48d9-bb60-c68da19edbca" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:42:31.661713 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.661651 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dce1805d-8afe-48d9-bb60-c68da19edbca" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:42:31.663235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.663212 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj" (OuterVolumeSpecName: "kube-api-access-9tghj") pod "dce1805d-8afe-48d9-bb60-c68da19edbca" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca"). InnerVolumeSpecName "kube-api-access-9tghj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:42:31.663304 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.663233 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dce1805d-8afe-48d9-bb60-c68da19edbca" (UID: "dce1805d-8afe-48d9-bb60-c68da19edbca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:42:31.762027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.761979 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:42:31.762027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.761998 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce1805d-8afe-48d9-bb60-c68da19edbca-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:42:31.762027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.762007 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tghj\" (UniqueName: \"kubernetes.io/projected/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-api-access-9tghj\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:42:31.762027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.762016 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dce1805d-8afe-48d9-bb60-c68da19edbca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:42:31.762027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:31.762024 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dce1805d-8afe-48d9-bb60-c68da19edbca-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:42:32.484478 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.484443 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" Sep 17 18:42:32.484899 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.484480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk" event={"ID":"dce1805d-8afe-48d9-bb60-c68da19edbca","Type":"ContainerDied","Data":"37fd7bacd5be6c0efa6e4b75a43d1d971632cb8dba89adb78177c2affa06eff2"} Sep 17 18:42:32.484899 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.484559 2559 scope.go:117] "RemoveContainer" containerID="abe9275ab8b524f09f5e24203ace79e19b5fd1897a0466070bd2ba35eeea81fd" Sep 17 18:42:32.493063 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.493046 2559 scope.go:117] "RemoveContainer" containerID="4f625421c218f3dd416c01a0ecf59e0731b52fd2f8dff4fef3525d2b8745d290" Sep 17 18:42:32.500891 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.500869 2559 scope.go:117] "RemoveContainer" containerID="5fd5ce5ea0bec9ef2733f540cdc59f1906aec0e69099784514cf0e0c81226b8c" Sep 17 18:42:32.507397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.507376 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:42:32.508900 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.508886 2559 scope.go:117] "RemoveContainer" containerID="8b3caae4a81b8bd8c41c655281da50fd6c3dd27eda2693c0438376190d98afdf" Sep 17 18:42:32.513347 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:32.513325 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6c65fb9bd7-f87fk"] Sep 17 18:42:33.991775 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:33.991741 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" path="/var/lib/kubelet/pods/dce1805d-8afe-48d9-bb60-c68da19edbca/volumes" Sep 17 18:42:34.471175 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:34.471145 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:42:34.471801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:34.471773 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:34.471991 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:34.471961 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:44.472314 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:44.472260 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:44.472790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:44.472767 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:42:54.472707 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:54.472666 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:54.473180 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:42:54.473080 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:43:04.472291 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:04.472243 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:43:04.472712 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:04.472690 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:43:14.472169 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:14.472128 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:43:14.472624 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:14.472603 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:43:24.472123 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:24.472077 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:43:24.472577 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:24.472552 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:43:34.471723 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:34.471681 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:43:34.472245 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:34.472133 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:43:44.472383 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:44.472346 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:43:44.472866 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:44.472668 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:43:56.609254 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.609167 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-6gkjn_52aba06a-ba4f-4ff9-9746-ffe3625ec084/kserve-container/0.log" Sep 17 18:43:56.755204 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.755171 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:43:56.755519 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.755489 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" containerID="cri-o://54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956" gracePeriod=30 Sep 17 18:43:56.755598 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.755534 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" containerID="cri-o://d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d" gracePeriod=30 Sep 17 18:43:56.755657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.755546 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" containerID="cri-o://0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5" gracePeriod=30 Sep 17 18:43:56.800840 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.800805 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:43:56.801050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.801021 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kserve-container" containerID="cri-o://3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" gracePeriod=30 Sep 17 18:43:56.801203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:56.801069 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kube-rbac-proxy" containerID="cri-o://80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" gracePeriod=30 Sep 17 18:43:57.031336 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.031310 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:43:57.121001 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.120970 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls\") pod \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " Sep 17 18:43:57.121162 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.121055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config\") pod \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " Sep 17 18:43:57.121162 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.121114 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pfj8\" (UniqueName: \"kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8\") pod \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\" (UID: \"52aba06a-ba4f-4ff9-9746-ffe3625ec084\") " Sep 17 18:43:57.121544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.121510 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52aba06a-ba4f-4ff9-9746-ffe3625ec084" (UID: "52aba06a-ba4f-4ff9-9746-ffe3625ec084"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:43:57.123059 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.123036 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8" (OuterVolumeSpecName: "kube-api-access-8pfj8") pod "52aba06a-ba4f-4ff9-9746-ffe3625ec084" (UID: "52aba06a-ba4f-4ff9-9746-ffe3625ec084"). InnerVolumeSpecName "kube-api-access-8pfj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:43:57.123258 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.123235 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52aba06a-ba4f-4ff9-9746-ffe3625ec084" (UID: "52aba06a-ba4f-4ff9-9746-ffe3625ec084"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:43:57.222171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.222118 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:43:57.222171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.222140 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pfj8\" (UniqueName: \"kubernetes.io/projected/52aba06a-ba4f-4ff9-9746-ffe3625ec084-kube-api-access-8pfj8\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:43:57.222171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.222151 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aba06a-ba4f-4ff9-9746-ffe3625ec084-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:43:57.864578 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864546 2559 generic.go:356] "Generic (PLEG): container finished" podID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerID="80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" exitCode=2 Sep 17 18:43:57.864578 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864575 2559 generic.go:356] "Generic (PLEG): container finished" podID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerID="3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" exitCode=2 Sep 17 18:43:57.865025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864593 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" Sep 17 18:43:57.865025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerDied","Data":"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3"} Sep 17 18:43:57.865025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerDied","Data":"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5"} Sep 17 18:43:57.865025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn" event={"ID":"52aba06a-ba4f-4ff9-9746-ffe3625ec084","Type":"ContainerDied","Data":"f2895eab40a68c47bdff50b24be384c8dc85b2bca9d6062321d1a285cb03f474"} Sep 17 18:43:57.865025 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.864768 2559 scope.go:117] "RemoveContainer" containerID="80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" Sep 17 18:43:57.867337 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.867315 2559 generic.go:356] "Generic (PLEG): container finished" podID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerID="0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5" exitCode=2 Sep 17 18:43:57.867466 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.867378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerDied","Data":"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5"} Sep 17 18:43:57.873711 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.873692 2559 scope.go:117] "RemoveContainer" containerID="3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" Sep 17 18:43:57.881590 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.881573 2559 scope.go:117] "RemoveContainer" containerID="80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" Sep 17 18:43:57.881844 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:43:57.881826 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3\": container with ID starting with 80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3 not found: ID does not exist" containerID="80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" Sep 17 18:43:57.881902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.881850 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3"} err="failed to get container status \"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3\": rpc error: code = NotFound desc = could not find container \"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3\": container with ID starting with 80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3 not found: ID does not exist" Sep 17 18:43:57.881902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.881866 2559 scope.go:117] "RemoveContainer" containerID="3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" Sep 17 18:43:57.882109 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:43:57.882093 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5\": container with ID starting with 3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5 not found: ID does not exist" containerID="3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" Sep 17 18:43:57.882148 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.882114 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5"} err="failed to get container status \"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5\": rpc error: code = NotFound desc = could not find container \"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5\": container with ID starting with 3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5 not found: ID does not exist" Sep 17 18:43:57.882148 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.882129 2559 scope.go:117] "RemoveContainer" containerID="80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3" Sep 17 18:43:57.882334 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.882316 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3"} err="failed to get container status \"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3\": rpc error: code = NotFound desc = could not find container \"80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3\": container with ID starting with 80f39795ebc69736e3bd6a7045efca7ed154427e4e297d6a7c28a5029fc97cf3 not found: ID does not exist" Sep 17 18:43:57.882334 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.882333 2559 scope.go:117] "RemoveContainer" containerID="3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5" Sep 17 18:43:57.882550 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.882535 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5"} err="failed to get container status \"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5\": rpc error: code = NotFound desc = could not find container \"3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5\": container with ID starting with 3b0af935468d80de12443f441237dbabb439d05e5b696d4b138a56d7302c3da5 not found: ID does not exist" Sep 17 18:43:57.886826 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.886804 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:43:57.890856 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.890811 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6gkjn"] Sep 17 18:43:57.992425 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:57.992402 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" path="/var/lib/kubelet/pods/52aba06a-ba4f-4ff9-9746-ffe3625ec084/volumes" Sep 17 18:43:59.467287 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:43:59.467242 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:00.883374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:00.883290 2559 generic.go:356] "Generic (PLEG): container finished" podID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerID="54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956" exitCode=0 Sep 17 18:44:00.883374 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:00.883343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerDied","Data":"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956"} Sep 17 18:44:04.466942 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:04.466905 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:04.472279 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:04.472253 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:44:04.472649 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:04.472623 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:44:09.467046 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.467008 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:09.467513 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.467120 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:44:09.920565 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.920535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:44:09.921895 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.921874 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:44:09.927977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.927957 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:44:09.929606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:09.929589 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:44:14.466902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:14.466862 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:14.472217 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:14.472192 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:44:14.472578 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:14.472553 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:44:19.466435 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:19.466394 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:24.466725 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:24.466682 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:44:24.472158 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:24.472126 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:44:24.472296 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:24.472253 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:44:24.472553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:24.472526 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:44:24.472662 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:24.472647 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:44:26.894307 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.894284 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:44:26.987563 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.987533 2559 generic.go:356] "Generic (PLEG): container finished" podID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerID="d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d" exitCode=0 Sep 17 18:44:26.987727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.987597 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" Sep 17 18:44:26.987727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.987618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerDied","Data":"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d"} Sep 17 18:44:26.987727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.987682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7" event={"ID":"165b0163-cdf6-4b33-9761-a8477ba0e39e","Type":"ContainerDied","Data":"d1e1d2c17c5e7f1ed9da8b1a418607b64ddf62ec1ac4360380aff977ebf43594"} Sep 17 18:44:26.987727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.987715 2559 scope.go:117] "RemoveContainer" containerID="d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d" Sep 17 18:44:26.995845 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:26.995829 2559 scope.go:117] "RemoveContainer" containerID="0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5" Sep 17 18:44:27.003407 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.003389 2559 scope.go:117] "RemoveContainer" containerID="54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956" Sep 17 18:44:27.010939 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.010920 2559 scope.go:117] "RemoveContainer" containerID="16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb" Sep 17 18:44:27.018376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.018356 2559 scope.go:117] "RemoveContainer" containerID="d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d" Sep 17 18:44:27.018693 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:44:27.018672 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d\": container with ID starting with d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d not found: ID does not exist" containerID="d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d" Sep 17 18:44:27.018754 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.018702 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d"} err="failed to get container status \"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d\": rpc error: code = NotFound desc = could not find container \"d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d\": container with ID starting with d2c1553657758430cd9d4a820745176721b9f772ff5f3585de6a8de42241f44d not found: ID does not exist" Sep 17 18:44:27.018754 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.018720 2559 scope.go:117] "RemoveContainer" containerID="0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5" Sep 17 18:44:27.018990 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:44:27.018964 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5\": container with ID starting with 0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5 not found: ID does not exist" containerID="0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5" Sep 17 18:44:27.019086 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.018999 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5"} err="failed to get container status \"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5\": rpc error: code = NotFound desc = could not find container \"0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5\": container with ID starting with 0fbed78a6f28079449c38dcfa604985463c5fb276900c4ce10587c2aaadaa0a5 not found: ID does not exist" Sep 17 18:44:27.019086 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.019015 2559 scope.go:117] "RemoveContainer" containerID="54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956" Sep 17 18:44:27.019225 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:44:27.019210 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956\": container with ID starting with 54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956 not found: ID does not exist" containerID="54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956" Sep 17 18:44:27.019260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.019228 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956"} err="failed to get container status \"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956\": rpc error: code = NotFound desc = could not find container \"54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956\": container with ID starting with 54d270e0db4c6f7f3e21275d0cc000cfe8cf1f1522b52e36418cfef69ee01956 not found: ID does not exist" Sep 17 18:44:27.019260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.019240 2559 scope.go:117] "RemoveContainer" containerID="16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb" Sep 17 18:44:27.019516 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:44:27.019491 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb\": container with ID starting with 16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb not found: ID does not exist" containerID="16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb" Sep 17 18:44:27.019612 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.019521 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb"} err="failed to get container status \"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb\": rpc error: code = NotFound desc = could not find container \"16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb\": container with ID starting with 16d91c26584a3e42eec48bd5cf9c412d584e45059cf1c8d8ee4ac5462d47b2cb not found: ID does not exist" Sep 17 18:44:27.053834 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.053794 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config\") pod \"165b0163-cdf6-4b33-9761-a8477ba0e39e\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " Sep 17 18:44:27.053992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.053906 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location\") pod \"165b0163-cdf6-4b33-9761-a8477ba0e39e\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " Sep 17 18:44:27.053992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.053930 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert\") pod \"165b0163-cdf6-4b33-9761-a8477ba0e39e\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " Sep 17 18:44:27.053992 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.053970 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls\") pod \"165b0163-cdf6-4b33-9761-a8477ba0e39e\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " Sep 17 18:44:27.054151 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.053998 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz58r\" (UniqueName: \"kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r\") pod \"165b0163-cdf6-4b33-9761-a8477ba0e39e\" (UID: \"165b0163-cdf6-4b33-9761-a8477ba0e39e\") " Sep 17 18:44:27.054265 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.054241 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "165b0163-cdf6-4b33-9761-a8477ba0e39e" (UID: "165b0163-cdf6-4b33-9761-a8477ba0e39e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:44:27.054327 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.054279 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "165b0163-cdf6-4b33-9761-a8477ba0e39e" (UID: "165b0163-cdf6-4b33-9761-a8477ba0e39e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:44:27.054327 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.054278 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "165b0163-cdf6-4b33-9761-a8477ba0e39e" (UID: "165b0163-cdf6-4b33-9761-a8477ba0e39e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:44:27.056148 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.056127 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "165b0163-cdf6-4b33-9761-a8477ba0e39e" (UID: "165b0163-cdf6-4b33-9761-a8477ba0e39e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:44:27.056245 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.056227 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r" (OuterVolumeSpecName: "kube-api-access-rz58r") pod "165b0163-cdf6-4b33-9761-a8477ba0e39e" (UID: "165b0163-cdf6-4b33-9761-a8477ba0e39e"). InnerVolumeSpecName "kube-api-access-rz58r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:44:27.155076 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.155042 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/165b0163-cdf6-4b33-9761-a8477ba0e39e-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:44:27.155076 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.155069 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:44:27.155076 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.155079 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/165b0163-cdf6-4b33-9761-a8477ba0e39e-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:44:27.155285 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.155086 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz58r\" (UniqueName: \"kubernetes.io/projected/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-api-access-rz58r\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:44:27.155285 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.155098 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/165b0163-cdf6-4b33-9761-a8477ba0e39e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:44:27.310302 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.310267 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:44:27.315179 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.315158 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d79fc8c86-kspg7"] Sep 17 18:44:27.993411 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:27.993380 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" path="/var/lib/kubelet/pods/165b0163-cdf6-4b33-9761-a8477ba0e39e/volumes" Sep 17 18:44:38.935904 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.935871 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936163 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936174 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936184 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936190 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936202 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="storage-initializer" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936207 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="storage-initializer" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936213 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936219 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kube-rbac-proxy" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936229 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936234 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936241 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936248 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936254 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936259 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936264 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936269 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936278 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="storage-initializer" Sep 17 18:44:38.936275 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936285 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="storage-initializer" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936291 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936296 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936339 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kserve-container" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936345 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kserve-container" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936352 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="kube-rbac-proxy" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936358 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="165b0163-cdf6-4b33-9761-a8477ba0e39e" containerName="agent" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936363 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="agent" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936368 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kserve-container" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936373 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52aba06a-ba4f-4ff9-9746-ffe3625ec084" containerName="kube-rbac-proxy" Sep 17 18:44:38.937124 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.936379 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce1805d-8afe-48d9-bb60-c68da19edbca" containerName="kube-rbac-proxy" Sep 17 18:44:38.940932 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.940916 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:38.943648 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.943626 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:44:38.943785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.943627 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:44:38.943785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.943667 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:44:38.943953 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.943886 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 18:44:38.944818 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.944797 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\"" Sep 17 18:44:38.944818 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.944812 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\"" Sep 17 18:44:38.944977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.944854 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:44:38.950642 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:38.950625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:44:39.041558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.041535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.041657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.041577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.041657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.041634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46txz\" (UniqueName: \"kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.041733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.041658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.041733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.041705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142273 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46txz\" (UniqueName: \"kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142362 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142583 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.142925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.142904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.143139 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.143117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.143139 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.143132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.144941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.144921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.151303 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.151287 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46txz\" (UniqueName: \"kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.252407 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.252344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:44:39.524469 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:39.524429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:44:40.042871 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:40.042835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerStarted","Data":"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf"} Sep 17 18:44:40.043340 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:40.042889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerStarted","Data":"f2bc5b803ddd82289442976c83526864dd72dbdc2344f493128f8ac59fadf074"} Sep 17 18:44:44.059864 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:44.059827 2559 generic.go:356] "Generic (PLEG): container finished" podID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerID="47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf" exitCode=0 Sep 17 18:44:44.060398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:44:44.059905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerDied","Data":"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf"} Sep 17 18:45:07.185768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:07.185686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerStarted","Data":"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31"} Sep 17 18:45:07.186194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:07.185799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerStarted","Data":"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c"} Sep 17 18:45:07.186194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:07.185977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:45:07.186194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:07.186114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:45:07.209883 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:07.209834 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podStartSLOduration=2.815766898 podStartE2EDuration="29.209819218s" totalImagesPullingTime="22.49204188s" totalInitContainerRuntime="3.90201044s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:07.207490279 +0000 UTC m=+959.196106528" watchObservedRunningTime="2026-09-17 18:45:07.209819218 +0000 UTC m=+959.198435467" Sep 17 18:45:13.195131 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:13.195104 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:45:43.195947 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:43.195910 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:45:53.196179 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:45:53.196141 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:46:03.196130 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:03.196081 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:46:13.196279 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:13.196237 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:46:23.751786 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:23.751751 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:46:29.038567 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:29.038535 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:46:29.039111 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:29.038829 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" containerID="cri-o://e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c" gracePeriod=30 Sep 17 18:46:29.039111 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:29.038899 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kube-rbac-proxy" containerID="cri-o://54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31" gracePeriod=30 Sep 17 18:46:29.529141 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:29.529099 2559 generic.go:356] "Generic (PLEG): container finished" podID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerID="54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31" exitCode=2 Sep 17 18:46:29.529329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:29.529170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerDied","Data":"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31"} Sep 17 18:46:33.190248 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.190199 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused" Sep 17 18:46:33.195641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.195611 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:46:33.776486 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.776436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:46:33.840102 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840039 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert\") pod \"8ccdf71f-8963-4697-a61b-b28f167ac09a\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " Sep 17 18:46:33.840102 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46txz\" (UniqueName: \"kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz\") pod \"8ccdf71f-8963-4697-a61b-b28f167ac09a\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " Sep 17 18:46:33.840102 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840099 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls\") pod \"8ccdf71f-8963-4697-a61b-b28f167ac09a\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " Sep 17 18:46:33.840278 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840131 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config\") pod \"8ccdf71f-8963-4697-a61b-b28f167ac09a\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " Sep 17 18:46:33.840278 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840153 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location\") pod \"8ccdf71f-8963-4697-a61b-b28f167ac09a\" (UID: \"8ccdf71f-8963-4697-a61b-b28f167ac09a\") " Sep 17 18:46:33.840498 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ccdf71f-8963-4697-a61b-b28f167ac09a" (UID: "8ccdf71f-8963-4697-a61b-b28f167ac09a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:46:33.840620 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840553 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ccdf71f-8963-4697-a61b-b28f167ac09a" (UID: "8ccdf71f-8963-4697-a61b-b28f167ac09a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:46:33.840620 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.840561 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ccdf71f-8963-4697-a61b-b28f167ac09a" (UID: "8ccdf71f-8963-4697-a61b-b28f167ac09a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:46:33.842192 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.842172 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ccdf71f-8963-4697-a61b-b28f167ac09a" (UID: "8ccdf71f-8963-4697-a61b-b28f167ac09a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:46:33.842277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.842245 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz" (OuterVolumeSpecName: "kube-api-access-46txz") pod "8ccdf71f-8963-4697-a61b-b28f167ac09a" (UID: "8ccdf71f-8963-4697-a61b-b28f167ac09a"). InnerVolumeSpecName "kube-api-access-46txz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:46:33.940829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.940796 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:46:33.940829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.940823 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46txz\" (UniqueName: \"kubernetes.io/projected/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-api-access-46txz\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:46:33.940829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.940834 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccdf71f-8963-4697-a61b-b28f167ac09a-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:46:33.941065 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.940843 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccdf71f-8963-4697-a61b-b28f167ac09a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:46:33.941065 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:33.940852 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccdf71f-8963-4697-a61b-b28f167ac09a-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:46:34.230034 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.229996 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230291 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230302 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230318 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="storage-initializer" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230324 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="storage-initializer" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230332 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kube-rbac-proxy" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230337 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kube-rbac-proxy" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230385 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kserve-container" Sep 17 18:46:34.230522 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.230392 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerName="kube-rbac-proxy" Sep 17 18:46:34.232592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.232576 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.235142 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.235118 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\"" Sep 17 18:46:34.235278 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.235141 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\"" Sep 17 18:46:34.242833 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.242807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.242965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.242843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.242965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.242868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77sls\" (UniqueName: \"kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.243063 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.242980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.243113 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.243062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.243733 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.243704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:46:34.344399 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.344621 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.344621 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.344621 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.344621 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:46:34.344586 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 17 18:46:34.344864 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77sls\" (UniqueName: \"kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.344945 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:46:34.344655 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls podName:24ce9740-2feb-410f-b3bc-0e577405baa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:46:34.844635466 +0000 UTC m=+1046.833251697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 17 18:46:34.344945 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.344809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.345104 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.345082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.345229 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.345207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.358224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.358195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77sls\" (UniqueName: \"kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.550220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.550126 2559 generic.go:356] "Generic (PLEG): container finished" podID="8ccdf71f-8963-4697-a61b-b28f167ac09a" containerID="e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c" exitCode=0 Sep 17 18:46:34.550220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.550171 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" Sep 17 18:46:34.550220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.550194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerDied","Data":"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c"} Sep 17 18:46:34.550482 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.550256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn" event={"ID":"8ccdf71f-8963-4697-a61b-b28f167ac09a","Type":"ContainerDied","Data":"f2bc5b803ddd82289442976c83526864dd72dbdc2344f493128f8ac59fadf074"} Sep 17 18:46:34.550482 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.550286 2559 scope.go:117] "RemoveContainer" containerID="54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31" Sep 17 18:46:34.558684 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.558663 2559 scope.go:117] "RemoveContainer" containerID="e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c" Sep 17 18:46:34.566139 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.566121 2559 scope.go:117] "RemoveContainer" containerID="47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf" Sep 17 18:46:34.573748 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.573720 2559 scope.go:117] "RemoveContainer" containerID="54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31" Sep 17 18:46:34.574141 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:46:34.574119 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31\": container with ID starting with 54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31 not found: ID does not exist" containerID="54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31" Sep 17 18:46:34.574227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.574151 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31"} err="failed to get container status \"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31\": rpc error: code = NotFound desc = could not find container \"54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31\": container with ID starting with 54fbe17327e3998317dc5b0fcb95dfffc5c17d7eeb3e358a14f7d9dd8a9e4c31 not found: ID does not exist" Sep 17 18:46:34.574227 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.574171 2559 scope.go:117] "RemoveContainer" containerID="e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c" Sep 17 18:46:34.574464 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:46:34.574427 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c\": container with ID starting with e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c not found: ID does not exist" containerID="e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c" Sep 17 18:46:34.574511 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.574469 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c"} err="failed to get container status \"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c\": rpc error: code = NotFound desc = could not find container \"e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c\": container with ID starting with e50bdc0ac96fee2f6685deae4e2830faa11155978be07b9da26234e64b6d961c not found: ID does not exist" Sep 17 18:46:34.574511 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.574487 2559 scope.go:117] "RemoveContainer" containerID="47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf" Sep 17 18:46:34.574727 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:46:34.574710 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf\": container with ID starting with 47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf not found: ID does not exist" containerID="47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf" Sep 17 18:46:34.574768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.574732 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf"} err="failed to get container status \"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf\": rpc error: code = NotFound desc = could not find container \"47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf\": container with ID starting with 47e3c9bbc9776987bb63519ef72b4f7d76c698897e417f226fd8bbdbefd755bf not found: ID does not exist" Sep 17 18:46:34.586434 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.586372 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:46:34.588776 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.588751 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zpptn"] Sep 17 18:46:34.849338 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.849228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:34.851666 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:34.851647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:35.143642 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:35.143602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:35.413060 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:35.413034 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:46:35.415066 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:46:35.415036 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ce9740_2feb_410f_b3bc_0e577405baa0.slice/crio-273fdc7bf98349584c08d60f36404a93a948abf3751c25e9529fa318ece9bed0 WatchSource:0}: Error finding container 273fdc7bf98349584c08d60f36404a93a948abf3751c25e9529fa318ece9bed0: Status 404 returned error can't find the container with id 273fdc7bf98349584c08d60f36404a93a948abf3751c25e9529fa318ece9bed0 Sep 17 18:46:35.561002 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:35.560958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerStarted","Data":"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848"} Sep 17 18:46:35.561179 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:35.561044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerStarted","Data":"273fdc7bf98349584c08d60f36404a93a948abf3751c25e9529fa318ece9bed0"} Sep 17 18:46:35.992704 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:35.992662 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ccdf71f-8963-4697-a61b-b28f167ac09a" path="/var/lib/kubelet/pods/8ccdf71f-8963-4697-a61b-b28f167ac09a/volumes" Sep 17 18:46:39.578910 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:39.578818 2559 generic.go:356] "Generic (PLEG): container finished" podID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerID="0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848" exitCode=0 Sep 17 18:46:39.578910 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:39.578878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerDied","Data":"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848"} Sep 17 18:46:40.586161 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:40.586127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerStarted","Data":"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19"} Sep 17 18:46:40.586558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:40.586188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerStarted","Data":"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f"} Sep 17 18:46:40.586558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:40.586386 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:40.610947 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:40.610903 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podStartSLOduration=2.807739036 podStartE2EDuration="6.610889123s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803150087s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:40.608208462 +0000 UTC m=+1052.596824711" watchObservedRunningTime="2026-09-17 18:46:40.610889123 +0000 UTC m=+1052.599505372" Sep 17 18:46:41.590225 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:41.590182 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:46:47.599576 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:46:47.599549 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:47:17.600565 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:47:17.600527 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:27.600706 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:47:27.600664 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:37.601009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:47:37.600970 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:47.600682 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:47:47.600640 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:58.150479 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:47:58.150431 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:48:04.338777 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:04.338735 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:48:04.339167 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:04.339063 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" containerID="cri-o://6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f" gracePeriod=30 Sep 17 18:48:04.339167 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:04.339098 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kube-rbac-proxy" containerID="cri-o://d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19" gracePeriod=30 Sep 17 18:48:04.936533 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:04.936498 2559 generic.go:356] "Generic (PLEG): container finished" podID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerID="d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19" exitCode=2 Sep 17 18:48:04.936708 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:04.936570 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerDied","Data":"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19"} Sep 17 18:48:07.595656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:07.595610 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Sep 17 18:48:07.601021 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:07.600995 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:09.273729 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.273704 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:48:09.388202 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388177 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77sls\" (UniqueName: \"kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls\") pod \"24ce9740-2feb-410f-b3bc-0e577405baa0\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " Sep 17 18:48:09.388350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388229 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location\") pod \"24ce9740-2feb-410f-b3bc-0e577405baa0\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " Sep 17 18:48:09.388350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388257 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config\") pod \"24ce9740-2feb-410f-b3bc-0e577405baa0\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " Sep 17 18:48:09.388350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388290 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") pod \"24ce9740-2feb-410f-b3bc-0e577405baa0\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " Sep 17 18:48:09.388350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert\") pod \"24ce9740-2feb-410f-b3bc-0e577405baa0\" (UID: \"24ce9740-2feb-410f-b3bc-0e577405baa0\") " Sep 17 18:48:09.388631 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388607 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24ce9740-2feb-410f-b3bc-0e577405baa0" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:48:09.388761 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24ce9740-2feb-410f-b3bc-0e577405baa0" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:48:09.388865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.388765 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24ce9740-2feb-410f-b3bc-0e577405baa0" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:48:09.390253 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.390234 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls" (OuterVolumeSpecName: "kube-api-access-77sls") pod "24ce9740-2feb-410f-b3bc-0e577405baa0" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0"). InnerVolumeSpecName "kube-api-access-77sls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:48:09.390322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.390300 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24ce9740-2feb-410f-b3bc-0e577405baa0" (UID: "24ce9740-2feb-410f-b3bc-0e577405baa0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:48:09.489906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.489871 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:48:09.489906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.489897 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ce9740-2feb-410f-b3bc-0e577405baa0-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:48:09.489906 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.489908 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ce9740-2feb-410f-b3bc-0e577405baa0-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:48:09.490127 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.489918 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77sls\" (UniqueName: \"kubernetes.io/projected/24ce9740-2feb-410f-b3bc-0e577405baa0-kube-api-access-77sls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:48:09.490127 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.489927 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ce9740-2feb-410f-b3bc-0e577405baa0-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:48:09.582192 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582164 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:48:09.582490 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582476 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" Sep 17 18:48:09.582541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582491 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" Sep 17 18:48:09.582541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582508 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kube-rbac-proxy" Sep 17 18:48:09.582541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582513 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kube-rbac-proxy" Sep 17 18:48:09.582541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582520 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="storage-initializer" Sep 17 18:48:09.582541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582526 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="storage-initializer" Sep 17 18:48:09.582716 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582575 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kube-rbac-proxy" Sep 17 18:48:09.582716 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.582586 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerName="kserve-container" Sep 17 18:48:09.584842 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.584828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.588766 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.588744 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\"" Sep 17 18:48:09.588766 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.588764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\"" Sep 17 18:48:09.601980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.601956 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:48:09.691941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.691832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.691941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.691892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.692174 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.691954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.692174 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.691991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.692174 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.692017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44xf\" (UniqueName: \"kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.792802 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.792768 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.792950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.792813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.792950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.792842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.792950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.792872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.792950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.792908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44xf\" (UniqueName: \"kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.793162 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:48:09.793019 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found Sep 17 18:48:09.793162 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:48:09.793088 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls podName:e854b1fb-384b-4fbf-b715-a8d164ea99ff nodeName:}" failed. No retries permitted until 2026-09-17 18:48:10.293067502 +0000 UTC m=+1142.281683734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found Sep 17 18:48:09.793279 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.793233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.793501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.793482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.793617 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.793599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.802668 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.802647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44xf\" (UniqueName: \"kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:09.960959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.960875 2559 generic.go:356] "Generic (PLEG): container finished" podID="24ce9740-2feb-410f-b3bc-0e577405baa0" containerID="6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f" exitCode=0 Sep 17 18:48:09.960959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.960941 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" Sep 17 18:48:09.960959 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.960951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerDied","Data":"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f"} Sep 17 18:48:09.961157 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.961016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx" event={"ID":"24ce9740-2feb-410f-b3bc-0e577405baa0","Type":"ContainerDied","Data":"273fdc7bf98349584c08d60f36404a93a948abf3751c25e9529fa318ece9bed0"} Sep 17 18:48:09.961157 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.961054 2559 scope.go:117] "RemoveContainer" containerID="d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19" Sep 17 18:48:09.970037 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.970016 2559 scope.go:117] "RemoveContainer" containerID="6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f" Sep 17 18:48:09.977303 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.977284 2559 scope.go:117] "RemoveContainer" containerID="0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848" Sep 17 18:48:09.985930 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.985909 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:48:09.986568 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.986536 2559 scope.go:117] "RemoveContainer" containerID="d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19" Sep 17 18:48:09.987237 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:48:09.987205 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19\": container with ID starting with d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19 not found: ID does not exist" containerID="d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19" Sep 17 18:48:09.987339 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.987247 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19"} err="failed to get container status \"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19\": rpc error: code = NotFound desc = could not find container \"d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19\": container with ID starting with d658a4ad45824039d03338d1e27425e48e630965147f813e67a6ca0f7874dd19 not found: ID does not exist" Sep 17 18:48:09.987339 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.987280 2559 scope.go:117] "RemoveContainer" containerID="6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f" Sep 17 18:48:09.987892 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:48:09.987867 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f\": container with ID starting with 6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f not found: ID does not exist" containerID="6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f" Sep 17 18:48:09.987975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.987901 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f"} err="failed to get container status \"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f\": rpc error: code = NotFound desc = could not find container \"6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f\": container with ID starting with 6e6be8adde83e9658a9eaafc3d935577bc6e4eda4563249cf6730326b9009e8f not found: ID does not exist" Sep 17 18:48:09.987975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.987926 2559 scope.go:117] "RemoveContainer" containerID="0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848" Sep 17 18:48:09.988278 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:48:09.988231 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848\": container with ID starting with 0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848 not found: ID does not exist" containerID="0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848" Sep 17 18:48:09.988376 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.988282 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848"} err="failed to get container status \"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848\": rpc error: code = NotFound desc = could not find container \"0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848\": container with ID starting with 0014442cc6ed2334cf94084187a7fc41bbf7a2a031a8257cf6f018be40cc7848 not found: ID does not exist" Sep 17 18:48:09.995791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:09.995771 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-clnwx"] Sep 17 18:48:10.297857 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.297773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:10.300234 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.300209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:10.495061 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.495019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:10.773223 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.773197 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:48:10.776204 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:48:10.776179 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode854b1fb_384b_4fbf_b715_a8d164ea99ff.slice/crio-8cef9e62c9558b16d1e2c3c29e3057c1495e4be88d1f8830442051f744f50ed4 WatchSource:0}: Error finding container 8cef9e62c9558b16d1e2c3c29e3057c1495e4be88d1f8830442051f744f50ed4: Status 404 returned error can't find the container with id 8cef9e62c9558b16d1e2c3c29e3057c1495e4be88d1f8830442051f744f50ed4 Sep 17 18:48:10.966247 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.966208 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerStarted","Data":"bb8ac063483fde62eb65443940854d89ee92ac7e4a34ab58b88b0891033afc9f"} Sep 17 18:48:10.966443 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:10.966271 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerStarted","Data":"8cef9e62c9558b16d1e2c3c29e3057c1495e4be88d1f8830442051f744f50ed4"} Sep 17 18:48:11.992018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:11.991981 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ce9740-2feb-410f-b3bc-0e577405baa0" path="/var/lib/kubelet/pods/24ce9740-2feb-410f-b3bc-0e577405baa0/volumes" Sep 17 18:48:14.983999 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:14.983967 2559 generic.go:356] "Generic (PLEG): container finished" podID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerID="bb8ac063483fde62eb65443940854d89ee92ac7e4a34ab58b88b0891033afc9f" exitCode=0 Sep 17 18:48:14.984391 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:14.984030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerDied","Data":"bb8ac063483fde62eb65443940854d89ee92ac7e4a34ab58b88b0891033afc9f"} Sep 17 18:48:15.991816 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:15.991784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerStarted","Data":"04e8edc1baed5988a591c6260986e2bd11d60482268a1665ce8e7ff3e8d3654c"} Sep 17 18:48:15.992269 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:15.991843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerStarted","Data":"74a0917318019caaf545d1ab9885329b7777185225700eee6c02edf2ca0c8941"} Sep 17 18:48:15.992269 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:15.992022 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:16.013146 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:16.013100 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podStartSLOduration=3.367082484 podStartE2EDuration="7.013085933s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.646003449s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:16.011617363 +0000 UTC m=+1148.000233611" watchObservedRunningTime="2026-09-17 18:48:16.013085933 +0000 UTC m=+1148.001702181" Sep 17 18:48:16.994952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:16.994914 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:23.004476 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:23.004385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:48:53.006163 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:48:53.006124 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 17 18:49:03.005691 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:03.005650 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 17 18:49:09.949367 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:09.949340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:49:09.949941 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:09.949914 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:49:09.958322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:09.958296 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:49:09.958917 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:09.958897 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:49:13.005293 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:13.005253 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 17 18:49:23.005977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:23.005930 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 17 18:49:33.457200 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:33.457165 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:49:39.774137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:39.774101 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:49:39.774588 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:39.774365 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" containerID="cri-o://74a0917318019caaf545d1ab9885329b7777185225700eee6c02edf2ca0c8941" gracePeriod=30 Sep 17 18:49:39.774785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:39.774721 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kube-rbac-proxy" containerID="cri-o://04e8edc1baed5988a591c6260986e2bd11d60482268a1665ce8e7ff3e8d3654c" gracePeriod=30 Sep 17 18:49:40.329736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:40.329704 2559 generic.go:356] "Generic (PLEG): container finished" podID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerID="04e8edc1baed5988a591c6260986e2bd11d60482268a1665ce8e7ff3e8d3654c" exitCode=2 Sep 17 18:49:40.329891 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:40.329767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerDied","Data":"04e8edc1baed5988a591c6260986e2bd11d60482268a1665ce8e7ff3e8d3654c"} Sep 17 18:49:42.998689 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:42.998652 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused" Sep 17 18:49:43.005269 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:43.005240 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 17 18:49:44.347877 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.347839 2559 generic.go:356] "Generic (PLEG): container finished" podID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerID="74a0917318019caaf545d1ab9885329b7777185225700eee6c02edf2ca0c8941" exitCode=0 Sep 17 18:49:44.348195 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.347914 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerDied","Data":"74a0917318019caaf545d1ab9885329b7777185225700eee6c02edf2ca0c8941"} Sep 17 18:49:44.419584 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.419562 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:49:44.433611 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433586 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert\") pod \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " Sep 17 18:49:44.433712 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433625 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config\") pod \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " Sep 17 18:49:44.433763 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433748 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") pod \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " Sep 17 18:49:44.433801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433787 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location\") pod \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " Sep 17 18:49:44.433836 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433808 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b44xf\" (UniqueName: \"kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf\") pod \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\" (UID: \"e854b1fb-384b-4fbf-b715-a8d164ea99ff\") " Sep 17 18:49:44.434081 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.433958 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e854b1fb-384b-4fbf-b715-a8d164ea99ff" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:49:44.434128 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.434070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e854b1fb-384b-4fbf-b715-a8d164ea99ff" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:49:44.434165 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.434122 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e854b1fb-384b-4fbf-b715-a8d164ea99ff" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:49:44.435829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.435800 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf" (OuterVolumeSpecName: "kube-api-access-b44xf") pod "e854b1fb-384b-4fbf-b715-a8d164ea99ff" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff"). InnerVolumeSpecName "kube-api-access-b44xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:49:44.435829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.435811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e854b1fb-384b-4fbf-b715-a8d164ea99ff" (UID: "e854b1fb-384b-4fbf-b715-a8d164ea99ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:49:44.535322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.535264 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e854b1fb-384b-4fbf-b715-a8d164ea99ff-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:49:44.535322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.535285 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:49:44.535322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.535296 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44xf\" (UniqueName: \"kubernetes.io/projected/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-api-access-b44xf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:49:44.535322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.535304 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:49:44.535322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:44.535312 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e854b1fb-384b-4fbf-b715-a8d164ea99ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:49:45.355261 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.355224 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" event={"ID":"e854b1fb-384b-4fbf-b715-a8d164ea99ff","Type":"ContainerDied","Data":"8cef9e62c9558b16d1e2c3c29e3057c1495e4be88d1f8830442051f744f50ed4"} Sep 17 18:49:45.355727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.355274 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr" Sep 17 18:49:45.355727 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.355292 2559 scope.go:117] "RemoveContainer" containerID="04e8edc1baed5988a591c6260986e2bd11d60482268a1665ce8e7ff3e8d3654c" Sep 17 18:49:45.364345 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.364325 2559 scope.go:117] "RemoveContainer" containerID="74a0917318019caaf545d1ab9885329b7777185225700eee6c02edf2ca0c8941" Sep 17 18:49:45.371515 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.371497 2559 scope.go:117] "RemoveContainer" containerID="bb8ac063483fde62eb65443940854d89ee92ac7e4a34ab58b88b0891033afc9f" Sep 17 18:49:45.386605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.386584 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:49:45.400042 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.400023 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-mn4hr"] Sep 17 18:49:45.992626 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:49:45.992592 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" path="/var/lib/kubelet/pods/e854b1fb-384b-4fbf-b715-a8d164ea99ff/volumes" Sep 17 18:50:02.581236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581157 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-27c1217c"] Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581458 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="storage-initializer" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581471 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="storage-initializer" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581490 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kube-rbac-proxy" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581496 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kube-rbac-proxy" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581504 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581509 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581558 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kserve-container" Sep 17 18:50:02.583636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.581568 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e854b1fb-384b-4fbf-b715-a8d164ea99ff" containerName="kube-rbac-proxy" Sep 17 18:50:02.584529 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.584511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:02.587307 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.587283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\"" Sep 17 18:50:02.587580 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.587556 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:50:02.588477 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.588440 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:50:02.590882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.590862 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-27c1217c"] Sep 17 18:50:02.667736 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.667693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf\") pod \"curl-metrics-27c1217c\" (UID: \"c709dc01-1bff-4a23-a51a-aef7ae93d5df\") " pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:02.769016 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.768986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf\") pod \"curl-metrics-27c1217c\" (UID: \"c709dc01-1bff-4a23-a51a-aef7ae93d5df\") " pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:02.779592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.779563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf\") pod \"curl-metrics-27c1217c\" (UID: \"c709dc01-1bff-4a23-a51a-aef7ae93d5df\") " pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:02.906837 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:02.906818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:03.178036 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:03.178006 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-27c1217c"] Sep 17 18:50:03.180563 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:50:03.180536 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc709dc01_1bff_4a23_a51a_aef7ae93d5df.slice/crio-0f4afa100e077e05afbd884b9cd838b068929ae912a971be84b4ebb45bdb161f WatchSource:0}: Error finding container 0f4afa100e077e05afbd884b9cd838b068929ae912a971be84b4ebb45bdb161f: Status 404 returned error can't find the container with id 0f4afa100e077e05afbd884b9cd838b068929ae912a971be84b4ebb45bdb161f Sep 17 18:50:03.182636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:03.182619 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:50:03.428968 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:03.428881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-27c1217c" event={"ID":"c709dc01-1bff-4a23-a51a-aef7ae93d5df","Type":"ContainerStarted","Data":"0f4afa100e077e05afbd884b9cd838b068929ae912a971be84b4ebb45bdb161f"} Sep 17 18:50:04.434357 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:04.434321 2559 generic.go:356] "Generic (PLEG): container finished" podID="c709dc01-1bff-4a23-a51a-aef7ae93d5df" containerID="f5f84aa0e0d872a55f1c591a8c8e3b1286836f0beff432f062ffcb96ec1a3da3" exitCode=0 Sep 17 18:50:04.434756 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:04.434412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-27c1217c" event={"ID":"c709dc01-1bff-4a23-a51a-aef7ae93d5df","Type":"ContainerDied","Data":"f5f84aa0e0d872a55f1c591a8c8e3b1286836f0beff432f062ffcb96ec1a3da3"} Sep 17 18:50:05.560384 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.560355 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:05.589442 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.589419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf\") pod \"c709dc01-1bff-4a23-a51a-aef7ae93d5df\" (UID: \"c709dc01-1bff-4a23-a51a-aef7ae93d5df\") " Sep 17 18:50:05.591578 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.591547 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf" (OuterVolumeSpecName: "kube-api-access-2l8sf") pod "c709dc01-1bff-4a23-a51a-aef7ae93d5df" (UID: "c709dc01-1bff-4a23-a51a-aef7ae93d5df"). InnerVolumeSpecName "kube-api-access-2l8sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:50:05.597829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.597809 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-27c1217c_c709dc01-1bff-4a23-a51a-aef7ae93d5df/curl-metrics-27c1217c/0.log" Sep 17 18:50:05.623085 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.623058 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-27c1217c"] Sep 17 18:50:05.629220 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.629196 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-27c1217c"] Sep 17 18:50:05.690899 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.690832 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/c709dc01-1bff-4a23-a51a-aef7ae93d5df-kube-api-access-2l8sf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:50:05.991987 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:05.991903 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c709dc01-1bff-4a23-a51a-aef7ae93d5df" path="/var/lib/kubelet/pods/c709dc01-1bff-4a23-a51a-aef7ae93d5df/volumes" Sep 17 18:50:06.443496 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:06.443472 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-27c1217c" Sep 17 18:50:06.443656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:06.443519 2559 scope.go:117] "RemoveContainer" containerID="f5f84aa0e0d872a55f1c591a8c8e3b1286836f0beff432f062ffcb96ec1a3da3" Sep 17 18:50:11.041397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.041357 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-9913fd75"] Sep 17 18:50:11.041791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.041673 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c709dc01-1bff-4a23-a51a-aef7ae93d5df" containerName="curl-metrics-27c1217c" Sep 17 18:50:11.041791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.041685 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c709dc01-1bff-4a23-a51a-aef7ae93d5df" containerName="curl-metrics-27c1217c" Sep 17 18:50:11.041791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.041752 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c709dc01-1bff-4a23-a51a-aef7ae93d5df" containerName="curl-metrics-27c1217c" Sep 17 18:50:11.045907 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.045890 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:11.048942 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.048919 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:50:11.049075 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.048982 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4mg9\"" Sep 17 18:50:11.049960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.049888 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:50:11.052155 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.052135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9913fd75"] Sep 17 18:50:11.132610 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.132582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskxr\" (UniqueName: \"kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr\") pod \"curl-metrics-9913fd75\" (UID: \"f29128bd-b439-45b7-852a-09364eb84a3a\") " pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:11.233063 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.233036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskxr\" (UniqueName: \"kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr\") pod \"curl-metrics-9913fd75\" (UID: \"f29128bd-b439-45b7-852a-09364eb84a3a\") " pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:11.243881 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.243858 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskxr\" (UniqueName: \"kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr\") pod \"curl-metrics-9913fd75\" (UID: \"f29128bd-b439-45b7-852a-09364eb84a3a\") " pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:11.364928 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.364848 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:11.631658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:11.631632 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9913fd75"] Sep 17 18:50:11.634154 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:50:11.634127 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf29128bd_b439_45b7_852a_09364eb84a3a.slice/crio-351376e4130057c0900192e8977ca0b7a2ee0eb15f3f539b07eb73404bb34380 WatchSource:0}: Error finding container 351376e4130057c0900192e8977ca0b7a2ee0eb15f3f539b07eb73404bb34380: Status 404 returned error can't find the container with id 351376e4130057c0900192e8977ca0b7a2ee0eb15f3f539b07eb73404bb34380 Sep 17 18:50:12.477241 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:12.477207 2559 generic.go:356] "Generic (PLEG): container finished" podID="f29128bd-b439-45b7-852a-09364eb84a3a" containerID="174a76ca1da7fa39ce7192dcf535cd6ad6c7b5e1f1bbce011393a39cb653a2ed" exitCode=0 Sep 17 18:50:12.477638 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:12.477292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9913fd75" event={"ID":"f29128bd-b439-45b7-852a-09364eb84a3a","Type":"ContainerDied","Data":"174a76ca1da7fa39ce7192dcf535cd6ad6c7b5e1f1bbce011393a39cb653a2ed"} Sep 17 18:50:12.477638 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:12.477332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9913fd75" event={"ID":"f29128bd-b439-45b7-852a-09364eb84a3a","Type":"ContainerStarted","Data":"351376e4130057c0900192e8977ca0b7a2ee0eb15f3f539b07eb73404bb34380"} Sep 17 18:50:13.602611 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.602588 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:13.625583 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.625558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-9913fd75_f29128bd-b439-45b7-852a-09364eb84a3a/curl-metrics-9913fd75/0.log" Sep 17 18:50:13.653014 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.652989 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-9913fd75"] Sep 17 18:50:13.653181 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.653160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskxr\" (UniqueName: \"kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr\") pod \"f29128bd-b439-45b7-852a-09364eb84a3a\" (UID: \"f29128bd-b439-45b7-852a-09364eb84a3a\") " Sep 17 18:50:13.655159 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.655131 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr" (OuterVolumeSpecName: "kube-api-access-jskxr") pod "f29128bd-b439-45b7-852a-09364eb84a3a" (UID: "f29128bd-b439-45b7-852a-09364eb84a3a"). InnerVolumeSpecName "kube-api-access-jskxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:50:13.658844 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.658822 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-9913fd75"] Sep 17 18:50:13.753952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.753879 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskxr\" (UniqueName: \"kubernetes.io/projected/f29128bd-b439-45b7-852a-09364eb84a3a-kube-api-access-jskxr\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:50:13.996413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:13.996373 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29128bd-b439-45b7-852a-09364eb84a3a" path="/var/lib/kubelet/pods/f29128bd-b439-45b7-852a-09364eb84a3a/volumes" Sep 17 18:50:14.487325 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:14.487290 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9913fd75" Sep 17 18:50:14.487549 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:14.487310 2559 scope.go:117] "RemoveContainer" containerID="174a76ca1da7fa39ce7192dcf535cd6ad6c7b5e1f1bbce011393a39cb653a2ed" Sep 17 18:50:19.058235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.058197 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:50:19.058627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.058572 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29128bd-b439-45b7-852a-09364eb84a3a" containerName="curl-metrics-9913fd75" Sep 17 18:50:19.058627 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.058586 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29128bd-b439-45b7-852a-09364eb84a3a" containerName="curl-metrics-9913fd75" Sep 17 18:50:19.058706 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.058640 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29128bd-b439-45b7-852a-09364eb84a3a" containerName="curl-metrics-9913fd75" Sep 17 18:50:19.063110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.063094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.065805 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.065778 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\"" Sep 17 18:50:19.065942 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.065834 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\"" Sep 17 18:50:19.066212 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.066188 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:50:19.066321 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.066256 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:50:19.067037 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.067020 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:50:19.067144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.067048 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 18:50:19.067144 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.067083 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:50:19.075286 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.075264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:50:19.198029 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.197989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt42h\" (UniqueName: \"kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.198029 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.198034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.198260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.198081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.198260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.198211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.198260 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.198234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.299530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.299491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.299530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.299536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.299790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.299668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt42h\" (UniqueName: \"kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.299790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.299707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.299943 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.299896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.300279 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.300258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.300393 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.300257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.300393 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.300364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.302126 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.302094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.309287 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.309230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt42h\" (UniqueName: \"kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h\") pod \"isvc-sklearn-predictor-84657b574f-7pjdw\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.372890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.372854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:19.648537 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:19.648494 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:50:19.652187 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:50:19.652138 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7ecec7_770a_42be_8575_337386269d98.slice/crio-00354ea683c6f46bf9fef7e5b7c3ffbe9bdaf6384331c74ba506ee3ae9f1135c WatchSource:0}: Error finding container 00354ea683c6f46bf9fef7e5b7c3ffbe9bdaf6384331c74ba506ee3ae9f1135c: Status 404 returned error can't find the container with id 00354ea683c6f46bf9fef7e5b7c3ffbe9bdaf6384331c74ba506ee3ae9f1135c Sep 17 18:50:20.515859 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:20.515820 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerStarted","Data":"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4"} Sep 17 18:50:20.516300 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:20.515912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerStarted","Data":"00354ea683c6f46bf9fef7e5b7c3ffbe9bdaf6384331c74ba506ee3ae9f1135c"} Sep 17 18:50:23.530969 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:23.530936 2559 generic.go:356] "Generic (PLEG): container finished" podID="af7ecec7-770a-42be-8575-337386269d98" containerID="53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4" exitCode=0 Sep 17 18:50:23.531331 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:23.530997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerDied","Data":"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4"} Sep 17 18:50:24.537561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.537525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerStarted","Data":"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3"} Sep 17 18:50:24.538010 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.537585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerStarted","Data":"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7"} Sep 17 18:50:24.538010 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.537881 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:24.538010 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.537937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:24.539314 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.539283 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:50:24.558680 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:24.558630 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podStartSLOduration=1.863135768 podStartE2EDuration="5.558614151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695478383s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:24.55703259 +0000 UTC m=+1276.545648838" watchObservedRunningTime="2026-09-17 18:50:24.558614151 +0000 UTC m=+1276.547230401" Sep 17 18:50:25.541686 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:25.541646 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:50:30.546782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:30.546753 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:50:30.547239 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:30.547213 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:50:40.547402 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:40.547365 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:50:50.548032 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:50:50.547990 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:51:00.547585 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:00.547541 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:51:10.547771 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:10.547725 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:51:20.547625 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:20.547582 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:51:30.547918 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:30.547877 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Sep 17 18:51:38.988311 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:38.988280 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:51:49.135620 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:49.135575 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:51:49.136016 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:49.135889 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" containerID="cri-o://da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7" gracePeriod=30 Sep 17 18:51:49.136016 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:49.135950 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kube-rbac-proxy" containerID="cri-o://38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3" gracePeriod=30 Sep 17 18:51:49.894535 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:49.894501 2559 generic.go:356] "Generic (PLEG): container finished" podID="af7ecec7-770a-42be-8575-337386269d98" containerID="38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3" exitCode=2 Sep 17 18:51:49.894701 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:49.894548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerDied","Data":"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3"} Sep 17 18:51:50.542699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:50.542655 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused" Sep 17 18:51:53.478521 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.478497 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:51:53.635623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635553 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert\") pod \"af7ecec7-770a-42be-8575-337386269d98\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " Sep 17 18:51:53.635623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt42h\" (UniqueName: \"kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h\") pod \"af7ecec7-770a-42be-8575-337386269d98\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " Sep 17 18:51:53.635822 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635660 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls\") pod \"af7ecec7-770a-42be-8575-337386269d98\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " Sep 17 18:51:53.635822 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635702 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location\") pod \"af7ecec7-770a-42be-8575-337386269d98\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " Sep 17 18:51:53.635822 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635744 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config\") pod \"af7ecec7-770a-42be-8575-337386269d98\" (UID: \"af7ecec7-770a-42be-8575-337386269d98\") " Sep 17 18:51:53.635965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635882 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af7ecec7-770a-42be-8575-337386269d98" (UID: "af7ecec7-770a-42be-8575-337386269d98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:51:53.636018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.635987 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:51:53.636058 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.636030 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af7ecec7-770a-42be-8575-337386269d98" (UID: "af7ecec7-770a-42be-8575-337386269d98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:51:53.636137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.636111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af7ecec7-770a-42be-8575-337386269d98" (UID: "af7ecec7-770a-42be-8575-337386269d98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:51:53.637690 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.637669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h" (OuterVolumeSpecName: "kube-api-access-gt42h") pod "af7ecec7-770a-42be-8575-337386269d98" (UID: "af7ecec7-770a-42be-8575-337386269d98"). InnerVolumeSpecName "kube-api-access-gt42h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:51:53.637776 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.637740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af7ecec7-770a-42be-8575-337386269d98" (UID: "af7ecec7-770a-42be-8575-337386269d98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:51:53.736527 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.736503 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ecec7-770a-42be-8575-337386269d98-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:51:53.736527 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.736524 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ecec7-770a-42be-8575-337386269d98-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:51:53.736669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.736534 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ecec7-770a-42be-8575-337386269d98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:51:53.736669 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.736545 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt42h\" (UniqueName: \"kubernetes.io/projected/af7ecec7-770a-42be-8575-337386269d98-kube-api-access-gt42h\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:51:53.912172 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.912136 2559 generic.go:356] "Generic (PLEG): container finished" podID="af7ecec7-770a-42be-8575-337386269d98" containerID="da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7" exitCode=0 Sep 17 18:51:53.912294 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.912206 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" Sep 17 18:51:53.912294 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.912232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerDied","Data":"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7"} Sep 17 18:51:53.912372 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.912303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw" event={"ID":"af7ecec7-770a-42be-8575-337386269d98","Type":"ContainerDied","Data":"00354ea683c6f46bf9fef7e5b7c3ffbe9bdaf6384331c74ba506ee3ae9f1135c"} Sep 17 18:51:53.912372 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.912342 2559 scope.go:117] "RemoveContainer" containerID="38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3" Sep 17 18:51:53.924013 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.922182 2559 scope.go:117] "RemoveContainer" containerID="da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7" Sep 17 18:51:53.930489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.930471 2559 scope.go:117] "RemoveContainer" containerID="53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4" Sep 17 18:51:53.937687 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.937669 2559 scope.go:117] "RemoveContainer" containerID="38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3" Sep 17 18:51:53.937954 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:51:53.937937 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3\": container with ID starting with 38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3 not found: ID does not exist" containerID="38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3" Sep 17 18:51:53.938006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.937962 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3"} err="failed to get container status \"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3\": rpc error: code = NotFound desc = could not find container \"38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3\": container with ID starting with 38fffa5e9aa72e9a6bc073bc3a88d1fef06a0f93025c14ec2e20c68a90f22aa3 not found: ID does not exist" Sep 17 18:51:53.938006 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.937978 2559 scope.go:117] "RemoveContainer" containerID="da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7" Sep 17 18:51:53.938237 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:51:53.938218 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7\": container with ID starting with da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7 not found: ID does not exist" containerID="da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7" Sep 17 18:51:53.938277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.938244 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7"} err="failed to get container status \"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7\": rpc error: code = NotFound desc = could not find container \"da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7\": container with ID starting with da70272a5144928a018062377bb6e12e33ba3765aa7d571db30d68ee3ea3ebd7 not found: ID does not exist" Sep 17 18:51:53.938277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.938259 2559 scope.go:117] "RemoveContainer" containerID="53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4" Sep 17 18:51:53.938504 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:51:53.938485 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4\": container with ID starting with 53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4 not found: ID does not exist" containerID="53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4" Sep 17 18:51:53.938553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.938510 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4"} err="failed to get container status \"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4\": rpc error: code = NotFound desc = could not find container \"53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4\": container with ID starting with 53ad5bc45639493d8a367568bd243e52978d8e737bdd4c9692b95c1310b20dd4 not found: ID does not exist" Sep 17 18:51:53.946715 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.946696 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:51:53.949137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.949118 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84657b574f-7pjdw"] Sep 17 18:51:53.991488 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:53.991440 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af7ecec7-770a-42be-8575-337386269d98" path="/var/lib/kubelet/pods/af7ecec7-770a-42be-8575-337386269d98/volumes" Sep 17 18:51:54.539619 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539548 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539892 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="storage-initializer" Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539905 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="storage-initializer" Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539932 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539939 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539947 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kube-rbac-proxy" Sep 17 18:51:54.539965 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.539952 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kube-rbac-proxy" Sep 17 18:51:54.540154 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.540001 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kserve-container" Sep 17 18:51:54.540154 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.540011 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ecec7-770a-42be-8575-337386269d98" containerName="kube-rbac-proxy" Sep 17 18:51:54.544487 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.544471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.547825 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.547804 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\"" Sep 17 18:51:54.548101 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.548084 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\"" Sep 17 18:51:54.548196 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.548102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:51:54.548196 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.548121 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:51:54.548389 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.548376 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:51:54.548956 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.548941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 18:51:54.549190 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.549177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:51:54.555101 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.555076 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:51:54.642135 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.642103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.642255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.642144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqcf\" (UniqueName: \"kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.642255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.642184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.642255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.642214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.642255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.642248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743057 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743134 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqcf\" (UniqueName: \"kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743134 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743194 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743863 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.743935 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.743879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.745598 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.745581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.752263 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.752237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqcf\" (UniqueName: \"kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:54.855599 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:54.855531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:51:55.132799 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:55.132774 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:51:55.135043 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:51:55.135017 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0eb7edb_e57b_4bee_b70c_3edfb80b2dd5.slice/crio-e958cd3151a27a9b67dd4c528859f7b00b7fa780cced4db4159578414bce95b9 WatchSource:0}: Error finding container e958cd3151a27a9b67dd4c528859f7b00b7fa780cced4db4159578414bce95b9: Status 404 returned error can't find the container with id e958cd3151a27a9b67dd4c528859f7b00b7fa780cced4db4159578414bce95b9 Sep 17 18:51:55.924525 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:55.924485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerStarted","Data":"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5"} Sep 17 18:51:55.924885 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:55.924546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerStarted","Data":"e958cd3151a27a9b67dd4c528859f7b00b7fa780cced4db4159578414bce95b9"} Sep 17 18:51:58.939330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:58.939300 2559 generic.go:356] "Generic (PLEG): container finished" podID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerID="c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5" exitCode=0 Sep 17 18:51:58.939674 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:51:58.939368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerDied","Data":"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5"} Sep 17 18:54:04.551940 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:04.551907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerStarted","Data":"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea"} Sep 17 18:54:05.558964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:05.558922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerStarted","Data":"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf"} Sep 17 18:54:05.559339 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:05.559043 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:54:05.582072 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:05.582028 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" podStartSLOduration=2.431110805 podStartE2EDuration="2m11.582013041s" totalImagesPullingTime="2m5.509548662s" totalInitContainerRuntime="3.641353574s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:05.580978454 +0000 UTC m=+1497.569594705" watchObservedRunningTime="2026-09-17 18:54:05.582013041 +0000 UTC m=+1497.570629290" Sep 17 18:54:06.562854 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:06.562825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:54:09.995108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:09.995077 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:54:09.995612 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:09.995077 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:54:10.011193 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:10.011162 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:54:10.011337 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:10.011166 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:54:12.571994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:12.571960 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:54:21.301141 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.301105 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:54:21.353259 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.353230 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:54:21.353475 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.353378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.358116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.358088 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\"" Sep 17 18:54:21.358263 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.358246 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Sep 17 18:54:21.360989 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.360966 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:54:21.363258 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.363242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.369794 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.369779 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Sep 17 18:54:21.380567 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.380545 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:54:21.521828 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.521828 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521826 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.522039 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvgp\" (UniqueName: \"kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.522039 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.522039 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.522039 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.521993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.522238 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.522050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds87q\" (UniqueName: \"kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.522238 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.522085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.522238 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.522132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.522238 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.522167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.622592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvgp\" (UniqueName: \"kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:54:21.622707 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds87q\" (UniqueName: \"kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:54:21.622788 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls podName:2a88fce9-2e45-48d0-815c-98da032c106a nodeName:}" failed. No retries permitted until 2026-09-17 18:54:22.122763527 +0000 UTC m=+1514.111379768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls") pod "isvc-canary-create-v2-predictor-77847bfbb9-p8p52" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a") : secret "isvc-canary-create-v2-predictor-serving-cert" not found Sep 17 18:54:21.622925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.622812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.623349 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623006 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.623422 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.623422 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.623566 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.623566 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.623659 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.623581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.625803 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.625779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.635355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.635323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvgp\" (UniqueName: \"kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp\") pod \"isvc-canary-create-predictor-66d699644d-fb48s\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.635497 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.635326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds87q\" (UniqueName: \"kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:21.672276 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.672249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:21.959548 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:21.959517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:54:21.959742 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:54:21.959719 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6727f915_c7d0_4a7b_8cb6_c9b58ff11499.slice/crio-4a3a683e4f7e7b334c2fbea897653b307971f2628cb732134b11ba9802e6334b WatchSource:0}: Error finding container 4a3a683e4f7e7b334c2fbea897653b307971f2628cb732134b11ba9802e6334b: Status 404 returned error can't find the container with id 4a3a683e4f7e7b334c2fbea897653b307971f2628cb732134b11ba9802e6334b Sep 17 18:54:22.128571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.128537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:22.130942 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.130896 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-p8p52\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:22.263369 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.263276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:22.550920 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.550867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:54:22.552014 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:54:22.551969 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a88fce9_2e45_48d0_815c_98da032c106a.slice/crio-dddd110eaa3678207279f4cefb6e3aff579ca69d3411b1b8c0ff313f01797861 WatchSource:0}: Error finding container dddd110eaa3678207279f4cefb6e3aff579ca69d3411b1b8c0ff313f01797861: Status 404 returned error can't find the container with id dddd110eaa3678207279f4cefb6e3aff579ca69d3411b1b8c0ff313f01797861 Sep 17 18:54:22.629753 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.629716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerStarted","Data":"dddd110eaa3678207279f4cefb6e3aff579ca69d3411b1b8c0ff313f01797861"} Sep 17 18:54:22.631243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.631219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerStarted","Data":"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35"} Sep 17 18:54:22.631335 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:22.631267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerStarted","Data":"4a3a683e4f7e7b334c2fbea897653b307971f2628cb732134b11ba9802e6334b"} Sep 17 18:54:23.638319 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:23.638283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerStarted","Data":"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075"} Sep 17 18:54:26.654468 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:26.654420 2559 generic.go:356] "Generic (PLEG): container finished" podID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerID="8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35" exitCode=0 Sep 17 18:54:26.654894 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:26.654483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerDied","Data":"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35"} Sep 17 18:54:26.656120 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:26.656100 2559 generic.go:356] "Generic (PLEG): container finished" podID="2a88fce9-2e45-48d0-815c-98da032c106a" containerID="ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075" exitCode=0 Sep 17 18:54:26.656218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:26.656169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerDied","Data":"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075"} Sep 17 18:54:27.662929 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.662891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerStarted","Data":"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a"} Sep 17 18:54:27.663425 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.662963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerStarted","Data":"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997"} Sep 17 18:54:27.663425 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.663259 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:27.663425 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.663416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:27.665107 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.665075 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:54:27.665236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.665215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerStarted","Data":"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00"} Sep 17 18:54:27.665301 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.665284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerStarted","Data":"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd"} Sep 17 18:54:27.665581 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.665562 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:27.665691 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.665607 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:27.666649 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.666627 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:54:27.682340 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.682298 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podStartSLOduration=2.738455765 podStartE2EDuration="6.682285829s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.943830064s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:27.680984757 +0000 UTC m=+1519.669601006" watchObservedRunningTime="2026-09-17 18:54:27.682285829 +0000 UTC m=+1519.670902077" Sep 17 18:54:27.701463 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:27.701410 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podStartSLOduration=3.047506186 podStartE2EDuration="6.701396753s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653890567s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:27.700338133 +0000 UTC m=+1519.688954381" watchObservedRunningTime="2026-09-17 18:54:27.701396753 +0000 UTC m=+1519.690013002" Sep 17 18:54:28.670553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:28.670509 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:54:28.671023 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:28.670667 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:54:33.675253 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:33.675219 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:54:33.675725 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:33.675557 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:54:33.675798 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:33.675769 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:54:33.676309 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:33.676282 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:54:42.649758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:42.649698 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Sep 17 18:54:43.676494 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:43.676434 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:54:43.676958 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:43.676433 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:54:52.575205 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:52.575169 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:54:53.676192 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:53.676147 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:54:53.676646 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:53.676295 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:54:54.482264 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:54.482232 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:54:54.482486 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:54.482460 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kserve-container" containerID="cri-o://f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea" gracePeriod=30 Sep 17 18:54:54.482560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:54.482496 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kube-rbac-proxy" containerID="cri-o://5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf" gracePeriod=30 Sep 17 18:54:54.793791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:54.793711 2559 generic.go:356] "Generic (PLEG): container finished" podID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerID="5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf" exitCode=2 Sep 17 18:54:54.794119 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:54.793784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerDied","Data":"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf"} Sep 17 18:54:57.568115 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:57.568071 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused" Sep 17 18:54:59.666215 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.666180 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:54:59.668871 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.668855 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.671636 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.671616 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\"" Sep 17 18:54:59.671772 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.671718 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\"" Sep 17 18:54:59.679561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.679538 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:54:59.741294 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.741270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.741413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.741302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z564l\" (UniqueName: \"kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.741413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.741333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.741413 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.741401 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.741553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.741433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.842691 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.842669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.842824 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.842702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z564l\" (UniqueName: \"kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.842824 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.842732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.842824 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.842780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.842973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.842827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.843228 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.843209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.843340 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.843322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.843506 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.843488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.845203 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.845185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.852306 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.852278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z564l\" (UniqueName: \"kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l\") pod \"isvc-sklearn-runtime-predictor-7c957758cc-tthj5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:54:59.979235 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:54:59.979148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:00.251924 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:00.251881 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:55:00.252638 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:55:00.252609 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b882ca_2fc4_4984_a9cf_44dde4171ed5.slice/crio-c6d667912de67046382d2b9475471fd7c572ed6c57d6841579545ba258b78c8c WatchSource:0}: Error finding container c6d667912de67046382d2b9475471fd7c572ed6c57d6841579545ba258b78c8c: Status 404 returned error can't find the container with id c6d667912de67046382d2b9475471fd7c572ed6c57d6841579545ba258b78c8c Sep 17 18:55:00.823346 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:00.823307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerStarted","Data":"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8"} Sep 17 18:55:00.823346 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:00.823372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerStarted","Data":"c6d667912de67046382d2b9475471fd7c572ed6c57d6841579545ba258b78c8c"} Sep 17 18:55:01.820316 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.820291 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:55:01.829747 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.829721 2559 generic.go:356] "Generic (PLEG): container finished" podID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerID="f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea" exitCode=0 Sep 17 18:55:01.830079 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.829769 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" Sep 17 18:55:01.830079 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.829806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerDied","Data":"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea"} Sep 17 18:55:01.830079 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.829858 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf" event={"ID":"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5","Type":"ContainerDied","Data":"e958cd3151a27a9b67dd4c528859f7b00b7fa780cced4db4159578414bce95b9"} Sep 17 18:55:01.830079 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.829887 2559 scope.go:117] "RemoveContainer" containerID="5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf" Sep 17 18:55:01.837963 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.837948 2559 scope.go:117] "RemoveContainer" containerID="f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea" Sep 17 18:55:01.845463 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.845433 2559 scope.go:117] "RemoveContainer" containerID="c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5" Sep 17 18:55:01.854668 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.854652 2559 scope.go:117] "RemoveContainer" containerID="5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf" Sep 17 18:55:01.855016 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:01.854992 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf\": container with ID starting with 5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf not found: ID does not exist" containerID="5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf" Sep 17 18:55:01.855083 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.855027 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf"} err="failed to get container status \"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf\": rpc error: code = NotFound desc = could not find container \"5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf\": container with ID starting with 5b0bb3ca554ad479cd66d7322cee108968a5fcf8e1c56076ac77f69fd034bfdf not found: ID does not exist" Sep 17 18:55:01.855083 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.855050 2559 scope.go:117] "RemoveContainer" containerID="f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea" Sep 17 18:55:01.855392 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:01.855359 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea\": container with ID starting with f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea not found: ID does not exist" containerID="f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea" Sep 17 18:55:01.855474 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.855405 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea"} err="failed to get container status \"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea\": rpc error: code = NotFound desc = could not find container \"f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea\": container with ID starting with f2fb0becf3dfe9d17dced4a850abe9d32f2db1e9bfdf75eb42c38442f359e8ea not found: ID does not exist" Sep 17 18:55:01.855474 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.855427 2559 scope.go:117] "RemoveContainer" containerID="c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5" Sep 17 18:55:01.855729 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:01.855706 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5\": container with ID starting with c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5 not found: ID does not exist" containerID="c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5" Sep 17 18:55:01.855787 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.855738 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5"} err="failed to get container status \"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5\": rpc error: code = NotFound desc = could not find container \"c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5\": container with ID starting with c4d2c590ab9f6a7a98cf794fa28930a848a32646ac84907e6df7fa12fa3076f5 not found: ID does not exist" Sep 17 18:55:01.962112 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962036 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert\") pod \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " Sep 17 18:55:01.962255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962114 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config\") pod \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " Sep 17 18:55:01.962255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962139 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qqcf\" (UniqueName: \"kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf\") pod \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " Sep 17 18:55:01.962255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962155 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location\") pod \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " Sep 17 18:55:01.962255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls\") pod \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\" (UID: \"b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5\") " Sep 17 18:55:01.962530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" (UID: "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:55:01.962605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" (UID: "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:55:01.962605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.962552 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" (UID: "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:55:01.964321 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.964299 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" (UID: "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:55:01.964422 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:01.964316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf" (OuterVolumeSpecName: "kube-api-access-4qqcf") pod "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" (UID: "b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5"). InnerVolumeSpecName "kube-api-access-4qqcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:55:02.062977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.062943 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:02.062977 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.062974 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:02.063150 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.062985 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqcf\" (UniqueName: \"kubernetes.io/projected/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kube-api-access-4qqcf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:02.063150 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.062995 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:02.063150 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.063004 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:02.150990 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.150952 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:55:02.153410 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:02.153386 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-xpdjf"] Sep 17 18:55:03.676009 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:03.675972 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:55:03.677603 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:03.676189 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:55:03.992870 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:03.992791 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" path="/var/lib/kubelet/pods/b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5/volumes" Sep 17 18:55:06.855618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:06.855525 2559 generic.go:356] "Generic (PLEG): container finished" podID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerID="a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8" exitCode=0 Sep 17 18:55:06.855618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:06.855601 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerDied","Data":"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8"} Sep 17 18:55:07.862657 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.862623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerStarted","Data":"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5"} Sep 17 18:55:07.863071 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.862680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerStarted","Data":"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967"} Sep 17 18:55:07.863071 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.862951 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:07.863178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.863119 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:07.864623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.864592 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 17 18:55:07.891247 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:07.891189 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podStartSLOduration=2.707403184 podStartE2EDuration="8.891171617s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.183768433s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:07.888676801 +0000 UTC m=+1559.877293050" watchObservedRunningTime="2026-09-17 18:55:07.891171617 +0000 UTC m=+1559.879787867" Sep 17 18:55:08.868399 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:08.868343 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 17 18:55:13.676135 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:13.676096 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:55:13.676579 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:13.676248 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:55:13.873347 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:13.873315 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:13.873842 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:13.873812 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 17 18:55:23.676397 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:23.676357 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:55:23.676814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:23.676368 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:55:23.874864 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:23.874834 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:31.801581 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:31.801545 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:55:31.801983 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:31.801798 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" containerID="cri-o://efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967" gracePeriod=30 Sep 17 18:55:31.801983 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:31.801825 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kube-rbac-proxy" containerID="cri-o://252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5" gracePeriod=30 Sep 17 18:55:31.980206 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:31.980178 2559 generic.go:356] "Generic (PLEG): container finished" podID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerID="252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5" exitCode=2 Sep 17 18:55:31.980345 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:31.980246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerDied","Data":"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5"} Sep 17 18:55:32.662255 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.662230 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:32.724719 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.724691 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location\") pod \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " Sep 17 18:55:32.724849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.724734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls\") pod \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " Sep 17 18:55:32.724849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.724753 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config\") pod \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " Sep 17 18:55:32.724849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.724805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert\") pod \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " Sep 17 18:55:32.724849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.724834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z564l\" (UniqueName: \"kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l\") pod \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\" (UID: \"34b882ca-2fc4-4984-a9cf-44dde4171ed5\") " Sep 17 18:55:32.725165 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.725139 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34b882ca-2fc4-4984-a9cf-44dde4171ed5" (UID: "34b882ca-2fc4-4984-a9cf-44dde4171ed5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:55:32.725236 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.725156 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34b882ca-2fc4-4984-a9cf-44dde4171ed5" (UID: "34b882ca-2fc4-4984-a9cf-44dde4171ed5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:55:32.726857 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.726831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34b882ca-2fc4-4984-a9cf-44dde4171ed5" (UID: "34b882ca-2fc4-4984-a9cf-44dde4171ed5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:55:32.726944 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.726928 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l" (OuterVolumeSpecName: "kube-api-access-z564l") pod "34b882ca-2fc4-4984-a9cf-44dde4171ed5" (UID: "34b882ca-2fc4-4984-a9cf-44dde4171ed5"). InnerVolumeSpecName "kube-api-access-z564l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:55:32.752159 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.752132 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34b882ca-2fc4-4984-a9cf-44dde4171ed5" (UID: "34b882ca-2fc4-4984-a9cf-44dde4171ed5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:55:32.825782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.825723 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z564l\" (UniqueName: \"kubernetes.io/projected/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-api-access-z564l\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:32.825782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.825745 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:32.825782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.825757 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b882ca-2fc4-4984-a9cf-44dde4171ed5-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:32.825782 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.825766 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:32.826176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.825775 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34b882ca-2fc4-4984-a9cf-44dde4171ed5-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:55:32.986318 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.986288 2559 generic.go:356] "Generic (PLEG): container finished" podID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerID="efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967" exitCode=0 Sep 17 18:55:32.986434 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.986344 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" Sep 17 18:55:32.986434 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.986372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerDied","Data":"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967"} Sep 17 18:55:32.986553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.986466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5" event={"ID":"34b882ca-2fc4-4984-a9cf-44dde4171ed5","Type":"ContainerDied","Data":"c6d667912de67046382d2b9475471fd7c572ed6c57d6841579545ba258b78c8c"} Sep 17 18:55:32.986553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.986512 2559 scope.go:117] "RemoveContainer" containerID="252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5" Sep 17 18:55:32.995710 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:32.995691 2559 scope.go:117] "RemoveContainer" containerID="efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967" Sep 17 18:55:33.003489 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.003473 2559 scope.go:117] "RemoveContainer" containerID="a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8" Sep 17 18:55:33.010185 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.010142 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:55:33.011371 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.011352 2559 scope.go:117] "RemoveContainer" containerID="252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5" Sep 17 18:55:33.011719 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:33.011695 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5\": container with ID starting with 252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5 not found: ID does not exist" containerID="252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5" Sep 17 18:55:33.011792 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.011728 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5"} err="failed to get container status \"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5\": rpc error: code = NotFound desc = could not find container \"252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5\": container with ID starting with 252b9c9baf119af0ab3c5cbea039a1eecf3879bed778b118add0118bcf96cfc5 not found: ID does not exist" Sep 17 18:55:33.011792 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.011750 2559 scope.go:117] "RemoveContainer" containerID="efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967" Sep 17 18:55:33.012031 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:33.012007 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967\": container with ID starting with efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967 not found: ID does not exist" containerID="efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967" Sep 17 18:55:33.012082 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.012040 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967"} err="failed to get container status \"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967\": rpc error: code = NotFound desc = could not find container \"efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967\": container with ID starting with efa70745bd030377511f49b251d16e9e32e1f6831831d48099febe1434920967 not found: ID does not exist" Sep 17 18:55:33.012082 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.012055 2559 scope.go:117] "RemoveContainer" containerID="a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8" Sep 17 18:55:33.012329 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:33.012313 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8\": container with ID starting with a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8 not found: ID does not exist" containerID="a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8" Sep 17 18:55:33.012400 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.012333 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8"} err="failed to get container status \"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8\": rpc error: code = NotFound desc = could not find container \"a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8\": container with ID starting with a3abf9dab814ccb8201dcfe3f09931e2116a23f5e4415e159fbbcabb9dc497b8 not found: ID does not exist" Sep 17 18:55:33.012472 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.012441 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c957758cc-tthj5"] Sep 17 18:55:33.676420 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.676385 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:55:33.676629 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.676387 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:55:33.992829 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:33.992754 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" path="/var/lib/kubelet/pods/34b882ca-2fc4-4984-a9cf-44dde4171ed5/volumes" Sep 17 18:55:36.831659 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.831621 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832087 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kserve-container" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832107 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kserve-container" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832119 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832128 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832149 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832158 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832181 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="storage-initializer" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832191 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="storage-initializer" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832201 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="storage-initializer" Sep 17 18:55:36.832210 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832208 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="storage-initializer" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832230 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832238 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832339 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832353 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kserve-container" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832364 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0eb7edb-e57b-4bee-b70c-3edfb80b2dd5" containerName="kube-rbac-proxy" Sep 17 18:55:36.832760 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.832374 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b882ca-2fc4-4984-a9cf-44dde4171ed5" containerName="kserve-container" Sep 17 18:55:36.835794 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.835772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.838401 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.838378 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\"" Sep 17 18:55:36.838517 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.838431 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\"" Sep 17 18:55:36.847262 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.847232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:55:36.856844 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.856820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.856980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.856858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzck\" (UniqueName: \"kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.856980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.856879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.856980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.856909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.856980 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.856959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958062 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzck\" (UniqueName: \"kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:36.958203 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found Sep 17 18:55:36.958516 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:55:36.958276 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls podName:48443fba-0dfd-4048-9492-79e134943d5e nodeName:}" failed. No retries permitted until 2026-09-17 18:55:37.458254886 +0000 UTC m=+1589.446871116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" (UID: "48443fba-0dfd-4048-9492-79e134943d5e") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found Sep 17 18:55:36.958638 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958822 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.958888 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.958871 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:36.967360 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:36.967343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzck\" (UniqueName: \"kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:37.461984 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:37.461952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:37.464309 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:37.464288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:37.746592 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:37.746496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:38.035997 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:38.035973 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:55:38.038491 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:55:38.038440 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48443fba_0dfd_4048_9492_79e134943d5e.slice/crio-6f6d406925b2a4034054c0a1bedd5fbeaaba7592e9c0374dc4726035e8b4bb71 WatchSource:0}: Error finding container 6f6d406925b2a4034054c0a1bedd5fbeaaba7592e9c0374dc4726035e8b4bb71: Status 404 returned error can't find the container with id 6f6d406925b2a4034054c0a1bedd5fbeaaba7592e9c0374dc4726035e8b4bb71 Sep 17 18:55:39.016994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:39.016963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerStarted","Data":"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce"} Sep 17 18:55:39.017192 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:39.017014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerStarted","Data":"6f6d406925b2a4034054c0a1bedd5fbeaaba7592e9c0374dc4726035e8b4bb71"} Sep 17 18:55:40.987737 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:40.987689 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 17 18:55:42.032712 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:42.032681 2559 generic.go:356] "Generic (PLEG): container finished" podID="48443fba-0dfd-4048-9492-79e134943d5e" containerID="166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce" exitCode=0 Sep 17 18:55:42.033073 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:42.032755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerDied","Data":"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce"} Sep 17 18:55:43.039681 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:43.039647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerStarted","Data":"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79"} Sep 17 18:55:43.040092 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:43.039707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerStarted","Data":"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168"} Sep 17 18:55:43.040092 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:43.039903 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:43.068110 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:43.068067 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podStartSLOduration=3.341964976 podStartE2EDuration="7.068052903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.726087927s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:43.065405609 +0000 UTC m=+1595.054021860" watchObservedRunningTime="2026-09-17 18:55:43.068052903 +0000 UTC m=+1595.056669231" Sep 17 18:55:43.677250 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:43.677215 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:55:44.043768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:44.043679 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:50.054427 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:50.054397 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:55:50.988269 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:55:50.988232 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:56:01.381890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.381859 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:56:01.382309 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.382098 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" containerID="cri-o://4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd" gracePeriod=30 Sep 17 18:56:01.382309 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.382148 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kube-rbac-proxy" containerID="cri-o://68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00" gracePeriod=30 Sep 17 18:56:01.399937 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.399913 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:56:01.400151 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.400132 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" containerID="cri-o://541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997" gracePeriod=30 Sep 17 18:56:01.400240 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:01.400219 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kube-rbac-proxy" containerID="cri-o://28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a" gracePeriod=30 Sep 17 18:56:02.131608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:02.131564 2559 generic.go:356] "Generic (PLEG): container finished" podID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerID="28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a" exitCode=2 Sep 17 18:56:02.131790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:02.131639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerDied","Data":"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a"} Sep 17 18:56:02.133876 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:02.133853 2559 generic.go:356] "Generic (PLEG): container finished" podID="2a88fce9-2e45-48d0-815c-98da032c106a" containerID="68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00" exitCode=2 Sep 17 18:56:02.133975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:02.133923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerDied","Data":"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00"} Sep 17 18:56:03.670676 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:03.670627 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Sep 17 18:56:03.671070 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:03.670627 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused" Sep 17 18:56:03.675994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:03.675975 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 17 18:56:06.338791 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.338769 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:56:06.395215 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395191 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert\") pod \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " Sep 17 18:56:06.395327 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395300 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls\") pod \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " Sep 17 18:56:06.395369 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssvgp\" (UniqueName: \"kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp\") pod \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " Sep 17 18:56:06.395412 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395388 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location\") pod \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " Sep 17 18:56:06.395460 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config\") pod \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\" (UID: \"6727f915-c7d0-4a7b-8cb6-c9b58ff11499\") " Sep 17 18:56:06.395566 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395522 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6727f915-c7d0-4a7b-8cb6-c9b58ff11499" (UID: "6727f915-c7d0-4a7b-8cb6-c9b58ff11499"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:06.395784 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6727f915-c7d0-4a7b-8cb6-c9b58ff11499" (UID: "6727f915-c7d0-4a7b-8cb6-c9b58ff11499"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:56:06.395899 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395817 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.395899 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.395868 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6727f915-c7d0-4a7b-8cb6-c9b58ff11499" (UID: "6727f915-c7d0-4a7b-8cb6-c9b58ff11499"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:06.397382 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.397359 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6727f915-c7d0-4a7b-8cb6-c9b58ff11499" (UID: "6727f915-c7d0-4a7b-8cb6-c9b58ff11499"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:56:06.397468 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.397399 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp" (OuterVolumeSpecName: "kube-api-access-ssvgp") pod "6727f915-c7d0-4a7b-8cb6-c9b58ff11499" (UID: "6727f915-c7d0-4a7b-8cb6-c9b58ff11499"). InnerVolumeSpecName "kube-api-access-ssvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:56:06.497074 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.497038 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.497074 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.497066 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvgp\" (UniqueName: \"kubernetes.io/projected/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-api-access-ssvgp\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.497254 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.497080 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.497254 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.497091 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6727f915-c7d0-4a7b-8cb6-c9b58ff11499-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.522068 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.522046 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:56:06.597348 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597322 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds87q\" (UniqueName: \"kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q\") pod \"2a88fce9-2e45-48d0-815c-98da032c106a\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " Sep 17 18:56:06.597531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597359 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert\") pod \"2a88fce9-2e45-48d0-815c-98da032c106a\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " Sep 17 18:56:06.597531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597389 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config\") pod \"2a88fce9-2e45-48d0-815c-98da032c106a\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " Sep 17 18:56:06.597531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597426 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location\") pod \"2a88fce9-2e45-48d0-815c-98da032c106a\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " Sep 17 18:56:06.597531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") pod \"2a88fce9-2e45-48d0-815c-98da032c106a\" (UID: \"2a88fce9-2e45-48d0-815c-98da032c106a\") " Sep 17 18:56:06.597802 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597767 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a88fce9-2e45-48d0-815c-98da032c106a" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:56:06.597802 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597780 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a88fce9-2e45-48d0-815c-98da032c106a" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:06.597802 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.597768 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a88fce9-2e45-48d0-815c-98da032c106a" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:06.599602 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.599569 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a88fce9-2e45-48d0-815c-98da032c106a" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:56:06.599693 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.599630 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q" (OuterVolumeSpecName: "kube-api-access-ds87q") pod "2a88fce9-2e45-48d0-815c-98da032c106a" (UID: "2a88fce9-2e45-48d0-815c-98da032c106a"). InnerVolumeSpecName "kube-api-access-ds87q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:56:06.608199 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608177 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:56:06.608511 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608498 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608512 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608525 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="storage-initializer" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608530 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="storage-initializer" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608542 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608548 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608557 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="storage-initializer" Sep 17 18:56:06.608560 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608562 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="storage-initializer" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608569 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608574 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608580 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608585 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608654 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608660 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kube-rbac-proxy" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608667 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" containerName="kserve-container" Sep 17 18:56:06.608768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.608673 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerName="kserve-container" Sep 17 18:56:06.611699 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.611685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.614928 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.614910 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Sep 17 18:56:06.615164 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.615149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Sep 17 18:56:06.628925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.628905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:56:06.698865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.698793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p28gf\" (UniqueName: \"kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.698865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.698825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.698865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.698843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.699052 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.698921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.699052 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.698982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.699052 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.699035 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds87q\" (UniqueName: \"kubernetes.io/projected/2a88fce9-2e45-48d0-815c-98da032c106a-kube-api-access-ds87q\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.699052 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.699051 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.699178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.699061 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88fce9-2e45-48d0-815c-98da032c106a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.699178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.699069 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88fce9-2e45-48d0-815c-98da032c106a-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.699178 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.699082 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88fce9-2e45-48d0-815c-98da032c106a-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:06.799421 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.799616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.799616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p28gf\" (UniqueName: \"kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.799616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.799616 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.800046 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.799994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.800322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.800294 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.800411 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.800306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.801937 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.801916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.812410 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.812392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p28gf\" (UniqueName: \"kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf\") pod \"isvc-canary-promote-predictor-dfdfd784d-gf2c5\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:06.921576 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:06.921540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:07.167809 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.167779 2559 generic.go:356] "Generic (PLEG): container finished" podID="2a88fce9-2e45-48d0-815c-98da032c106a" containerID="4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd" exitCode=0 Sep 17 18:56:07.167964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.167833 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" Sep 17 18:56:07.167964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.167845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerDied","Data":"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd"} Sep 17 18:56:07.167964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.167917 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52" event={"ID":"2a88fce9-2e45-48d0-815c-98da032c106a","Type":"ContainerDied","Data":"dddd110eaa3678207279f4cefb6e3aff579ca69d3411b1b8c0ff313f01797861"} Sep 17 18:56:07.167964 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.167949 2559 scope.go:117] "RemoveContainer" containerID="68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00" Sep 17 18:56:07.177137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.177104 2559 generic.go:356] "Generic (PLEG): container finished" podID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" containerID="541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997" exitCode=0 Sep 17 18:56:07.177137 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.177152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerDied","Data":"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997"} Sep 17 18:56:07.177382 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.177245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" event={"ID":"6727f915-c7d0-4a7b-8cb6-c9b58ff11499","Type":"ContainerDied","Data":"4a3a683e4f7e7b334c2fbea897653b307971f2628cb732134b11ba9802e6334b"} Sep 17 18:56:07.177382 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.177341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s" Sep 17 18:56:07.186140 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.185671 2559 scope.go:117] "RemoveContainer" containerID="4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd" Sep 17 18:56:07.193753 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.193737 2559 scope.go:117] "RemoveContainer" containerID="ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075" Sep 17 18:56:07.200624 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.200600 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:56:07.201826 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.201705 2559 scope.go:117] "RemoveContainer" containerID="68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00" Sep 17 18:56:07.202095 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.202066 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00\": container with ID starting with 68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00 not found: ID does not exist" containerID="68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00" Sep 17 18:56:07.202180 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202104 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00"} err="failed to get container status \"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00\": rpc error: code = NotFound desc = could not find container \"68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00\": container with ID starting with 68cc1d82cc1890a4e7f9df980bbc9080de11c6d1436156149eea566368409e00 not found: ID does not exist" Sep 17 18:56:07.202180 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202127 2559 scope.go:117] "RemoveContainer" containerID="4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd" Sep 17 18:56:07.202493 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.202440 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd\": container with ID starting with 4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd not found: ID does not exist" containerID="4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd" Sep 17 18:56:07.202605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202503 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd"} err="failed to get container status \"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd\": rpc error: code = NotFound desc = could not find container \"4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd\": container with ID starting with 4084e52db0261febbe3752c3675446b250cc5854f1e1a2a0da052e57ff73ffcd not found: ID does not exist" Sep 17 18:56:07.202605 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202527 2559 scope.go:117] "RemoveContainer" containerID="ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075" Sep 17 18:56:07.202836 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.202809 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075\": container with ID starting with ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075 not found: ID does not exist" containerID="ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075" Sep 17 18:56:07.202882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202843 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075"} err="failed to get container status \"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075\": rpc error: code = NotFound desc = could not find container \"ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075\": container with ID starting with ac0ea4802f22881efeb50fc7d17ba8960919fb26383282eb01b88d122997b075 not found: ID does not exist" Sep 17 18:56:07.202882 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.202860 2559 scope.go:117] "RemoveContainer" containerID="28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a" Sep 17 18:56:07.204050 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.204031 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:56:07.205795 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:56:07.205767 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc67759_758d_4876_8ded_d6236e180be9.slice/crio-a5d8f0d07bdf89017c264e8f6f38257926499716422e220ceeef4a4d1c9a1615 WatchSource:0}: Error finding container a5d8f0d07bdf89017c264e8f6f38257926499716422e220ceeef4a4d1c9a1615: Status 404 returned error can't find the container with id a5d8f0d07bdf89017c264e8f6f38257926499716422e220ceeef4a4d1c9a1615 Sep 17 18:56:07.206321 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.206299 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-p8p52"] Sep 17 18:56:07.215184 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.215159 2559 scope.go:117] "RemoveContainer" containerID="541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997" Sep 17 18:56:07.218910 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.218886 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:56:07.223330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.223309 2559 scope.go:117] "RemoveContainer" containerID="8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35" Sep 17 18:56:07.224488 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.224468 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-fb48s"] Sep 17 18:56:07.234555 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.234539 2559 scope.go:117] "RemoveContainer" containerID="28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a" Sep 17 18:56:07.234831 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.234815 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a\": container with ID starting with 28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a not found: ID does not exist" containerID="28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a" Sep 17 18:56:07.234902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.234840 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a"} err="failed to get container status \"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a\": rpc error: code = NotFound desc = could not find container \"28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a\": container with ID starting with 28d376367cdc911ed468ae1c456e9e8e17e6703e92f60a91d2ec8100bd78290a not found: ID does not exist" Sep 17 18:56:07.234902 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.234862 2559 scope.go:117] "RemoveContainer" containerID="541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997" Sep 17 18:56:07.235127 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.235110 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997\": container with ID starting with 541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997 not found: ID does not exist" containerID="541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997" Sep 17 18:56:07.235162 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.235132 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997"} err="failed to get container status \"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997\": rpc error: code = NotFound desc = could not find container \"541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997\": container with ID starting with 541cab407c6d9862ca5dc3df21d76f3514ad9d3dc67fd068b93d5bc3b87cd997 not found: ID does not exist" Sep 17 18:56:07.235162 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.235147 2559 scope.go:117] "RemoveContainer" containerID="8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35" Sep 17 18:56:07.235372 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:07.235354 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35\": container with ID starting with 8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35 not found: ID does not exist" containerID="8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35" Sep 17 18:56:07.235427 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.235378 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35"} err="failed to get container status \"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35\": rpc error: code = NotFound desc = could not find container \"8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35\": container with ID starting with 8fff60ca5e90ad9b92f03c9c9f414e19298ae1a6b03cd096b37e89af0aecaa35 not found: ID does not exist" Sep 17 18:56:07.992773 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.992738 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a88fce9-2e45-48d0-815c-98da032c106a" path="/var/lib/kubelet/pods/2a88fce9-2e45-48d0-815c-98da032c106a/volumes" Sep 17 18:56:07.993240 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:07.993227 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6727f915-c7d0-4a7b-8cb6-c9b58ff11499" path="/var/lib/kubelet/pods/6727f915-c7d0-4a7b-8cb6-c9b58ff11499/volumes" Sep 17 18:56:08.184834 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:08.184797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerStarted","Data":"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19"} Sep 17 18:56:08.185011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:08.184866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerStarted","Data":"a5d8f0d07bdf89017c264e8f6f38257926499716422e220ceeef4a4d1c9a1615"} Sep 17 18:56:11.200010 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:11.199980 2559 generic.go:356] "Generic (PLEG): container finished" podID="8cc67759-758d-4876-8ded-d6236e180be9" containerID="62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19" exitCode=0 Sep 17 18:56:11.200350 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:11.200050 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerDied","Data":"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19"} Sep 17 18:56:12.208403 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:12.208365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerStarted","Data":"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef"} Sep 17 18:56:12.208843 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:12.208436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerStarted","Data":"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6"} Sep 17 18:56:12.208843 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:12.208648 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:12.230433 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:12.230382 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podStartSLOduration=2.544480246 podStartE2EDuration="6.23036701s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685886764s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:12.228423262 +0000 UTC m=+1624.217039513" watchObservedRunningTime="2026-09-17 18:56:12.23036701 +0000 UTC m=+1624.218983312" Sep 17 18:56:13.213506 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:13.213474 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:13.214646 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:13.214620 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:14.218055 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:14.218017 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:19.222936 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:19.222905 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:56:19.223440 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:19.223406 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:20.149503 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:20.149440 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Sep 17 18:56:29.223994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:29.223947 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:30.057501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:30.057470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:56:36.907544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:36.907513 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:56:36.907914 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:36.907741 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" containerID="cri-o://0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168" gracePeriod=30 Sep 17 18:56:36.907914 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:36.907812 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" containerID="cri-o://b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79" gracePeriod=30 Sep 17 18:56:37.312597 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:37.312512 2559 generic.go:356] "Generic (PLEG): container finished" podID="48443fba-0dfd-4048-9492-79e134943d5e" containerID="b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79" exitCode=2 Sep 17 18:56:37.312597 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:37.312558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerDied","Data":"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79"} Sep 17 18:56:39.223950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:39.223908 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:40.049581 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:40.049533 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Sep 17 18:56:41.096620 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:41.096573 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 18:56:42.130677 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.130638 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:56:42.134329 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.134305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.137494 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.137472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\"" Sep 17 18:56:42.137609 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.137478 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\"" Sep 17 18:56:42.147961 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.147931 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:56:42.199943 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.199911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.200108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.199950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.200108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.199979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.200108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.200040 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.200108 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.200078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hnc\" (UniqueName: \"kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.301584 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.301548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hnc\" (UniqueName: \"kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.301778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.301613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.301778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.301640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.301778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.301673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.301778 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.301701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.302116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.302090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.302398 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.302364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.302527 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.302435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.304258 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.304239 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.312092 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.312068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hnc\" (UniqueName: \"kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc\") pod \"isvc-sklearn-v2-predictor-7df954dcd9-v5vwp\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.443880 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.443856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:42.734871 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:42.734847 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:56:42.737596 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:56:42.737569 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1ef929_5afc_4b9a_a2b8_f6b256e1b1c3.slice/crio-81651163d3f53a4077ba3dc53238b0727ec063f88633697e93e07d06454886f3 WatchSource:0}: Error finding container 81651163d3f53a4077ba3dc53238b0727ec063f88633697e93e07d06454886f3: Status 404 returned error can't find the container with id 81651163d3f53a4077ba3dc53238b0727ec063f88633697e93e07d06454886f3 Sep 17 18:56:43.340158 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:43.340122 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerStarted","Data":"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64"} Sep 17 18:56:43.340550 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:43.340183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerStarted","Data":"81651163d3f53a4077ba3dc53238b0727ec063f88633697e93e07d06454886f3"} Sep 17 18:56:45.049266 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:45.049219 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Sep 17 18:56:47.051835 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.051810 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:56:47.152655 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.152627 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config\") pod \"48443fba-0dfd-4048-9492-79e134943d5e\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " Sep 17 18:56:47.152830 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.152682 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfzck\" (UniqueName: \"kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck\") pod \"48443fba-0dfd-4048-9492-79e134943d5e\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " Sep 17 18:56:47.152830 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.152702 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert\") pod \"48443fba-0dfd-4048-9492-79e134943d5e\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " Sep 17 18:56:47.152830 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.152735 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location\") pod \"48443fba-0dfd-4048-9492-79e134943d5e\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " Sep 17 18:56:47.152830 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.152759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") pod \"48443fba-0dfd-4048-9492-79e134943d5e\" (UID: \"48443fba-0dfd-4048-9492-79e134943d5e\") " Sep 17 18:56:47.153109 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.153079 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48443fba-0dfd-4048-9492-79e134943d5e" (UID: "48443fba-0dfd-4048-9492-79e134943d5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:47.153172 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.153093 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48443fba-0dfd-4048-9492-79e134943d5e" (UID: "48443fba-0dfd-4048-9492-79e134943d5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:56:47.153172 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.153102 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48443fba-0dfd-4048-9492-79e134943d5e" (UID: "48443fba-0dfd-4048-9492-79e134943d5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:56:47.154762 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.154734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48443fba-0dfd-4048-9492-79e134943d5e" (UID: "48443fba-0dfd-4048-9492-79e134943d5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:56:47.154865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.154815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck" (OuterVolumeSpecName: "kube-api-access-sfzck") pod "48443fba-0dfd-4048-9492-79e134943d5e" (UID: "48443fba-0dfd-4048-9492-79e134943d5e"). InnerVolumeSpecName "kube-api-access-sfzck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:56:47.254353 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.254328 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:47.254353 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.254350 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzck\" (UniqueName: \"kubernetes.io/projected/48443fba-0dfd-4048-9492-79e134943d5e-kube-api-access-sfzck\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:47.254530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.254360 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48443fba-0dfd-4048-9492-79e134943d5e-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:47.254530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.254369 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48443fba-0dfd-4048-9492-79e134943d5e-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:47.254530 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.254377 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48443fba-0dfd-4048-9492-79e134943d5e-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:56:47.361689 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.361659 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerID="623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64" exitCode=0 Sep 17 18:56:47.361836 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.361728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerDied","Data":"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64"} Sep 17 18:56:47.363872 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.363848 2559 generic.go:356] "Generic (PLEG): container finished" podID="48443fba-0dfd-4048-9492-79e134943d5e" containerID="0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168" exitCode=0 Sep 17 18:56:47.363978 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.363910 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" Sep 17 18:56:47.363978 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.363928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerDied","Data":"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168"} Sep 17 18:56:47.364096 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.363996 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n" event={"ID":"48443fba-0dfd-4048-9492-79e134943d5e","Type":"ContainerDied","Data":"6f6d406925b2a4034054c0a1bedd5fbeaaba7592e9c0374dc4726035e8b4bb71"} Sep 17 18:56:47.364096 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.364039 2559 scope.go:117] "RemoveContainer" containerID="b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79" Sep 17 18:56:47.377306 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.377287 2559 scope.go:117] "RemoveContainer" containerID="0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168" Sep 17 18:56:47.392494 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.392472 2559 scope.go:117] "RemoveContainer" containerID="166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce" Sep 17 18:56:47.398515 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.398494 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:56:47.403183 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.403138 2559 scope.go:117] "RemoveContainer" containerID="b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79" Sep 17 18:56:47.403525 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:47.403498 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79\": container with ID starting with b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79 not found: ID does not exist" containerID="b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79" Sep 17 18:56:47.403624 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.403535 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79"} err="failed to get container status \"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79\": rpc error: code = NotFound desc = could not find container \"b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79\": container with ID starting with b940dda6d29aaa8787b4334876de28a7f976f3253056e544fdc1f647ed185f79 not found: ID does not exist" Sep 17 18:56:47.403624 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.403557 2559 scope.go:117] "RemoveContainer" containerID="0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168" Sep 17 18:56:47.403878 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:47.403854 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168\": container with ID starting with 0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168 not found: ID does not exist" containerID="0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168" Sep 17 18:56:47.403951 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.403887 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168"} err="failed to get container status \"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168\": rpc error: code = NotFound desc = could not find container \"0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168\": container with ID starting with 0a931caf16888dc469b47d66dae5ddd84561abfae1cec1598c7fb252f37b0168 not found: ID does not exist" Sep 17 18:56:47.403951 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.403904 2559 scope.go:117] "RemoveContainer" containerID="166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce" Sep 17 18:56:47.404257 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:56:47.404234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce\": container with ID starting with 166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce not found: ID does not exist" containerID="166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce" Sep 17 18:56:47.404326 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.404264 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce"} err="failed to get container status \"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce\": rpc error: code = NotFound desc = could not find container \"166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce\": container with ID starting with 166977d6283e41f87b79900f301b7f77e9685bc0d2f8c227206df9b5edf9f6ce not found: ID does not exist" Sep 17 18:56:47.404559 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.404542 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-lp29n"] Sep 17 18:56:47.992769 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:47.992737 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48443fba-0dfd-4048-9492-79e134943d5e" path="/var/lib/kubelet/pods/48443fba-0dfd-4048-9492-79e134943d5e/volumes" Sep 17 18:56:48.371680 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:48.371587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerStarted","Data":"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0"} Sep 17 18:56:48.371680 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:48.371657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerStarted","Data":"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9"} Sep 17 18:56:48.372171 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:48.371906 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:48.392890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:48.392841 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podStartSLOduration=2.825856393 podStartE2EDuration="6.392826341s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.566969948s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:48.390762826 +0000 UTC m=+1660.379379076" watchObservedRunningTime="2026-09-17 18:56:48.392826341 +0000 UTC m=+1660.381442655" Sep 17 18:56:49.223532 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:49.223495 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:56:49.377322 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:49.377284 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:49.378737 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:49.378708 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:56:50.382276 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:50.382228 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:56:55.386508 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:55.386479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:56:55.387100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:55.387074 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:56:59.223379 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:56:59.223331 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:57:05.387741 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:05.387700 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:09.224066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:09.224025 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:57:15.387796 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:15.387751 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:19.223934 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:19.223899 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:57:25.387505 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:25.387398 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:29.224369 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:29.224339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:57:35.387682 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:35.387643 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:36.578470 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.578418 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:57:36.579018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.578720 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" containerID="cri-o://744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6" gracePeriod=30 Sep 17 18:57:36.579018 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.578765 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kube-rbac-proxy" containerID="cri-o://4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef" gracePeriod=30 Sep 17 18:57:36.628854 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.628807 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:57:36.629176 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629164 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" Sep 17 18:57:36.629224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629178 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" Sep 17 18:57:36.629224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629193 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="storage-initializer" Sep 17 18:57:36.629224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629199 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="storage-initializer" Sep 17 18:57:36.629224 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629211 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" Sep 17 18:57:36.629355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629218 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" Sep 17 18:57:36.629355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629294 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kserve-container" Sep 17 18:57:36.629355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.629303 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48443fba-0dfd-4048-9492-79e134943d5e" containerName="kube-rbac-proxy" Sep 17 18:57:36.632487 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.632472 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.635263 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.635241 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Sep 17 18:57:36.646533 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.646512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:57:36.760651 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.760619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.760801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.760661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.760801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.760682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.760801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.760707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.760801 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.760731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrf9m\" (UniqueName: \"kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.861960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.861881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.861960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.861916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.861960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.861938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.861960 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.861960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrf9m\" (UniqueName: \"kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.862276 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.862048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.862428 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.862395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.862575 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.862555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.862680 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.862662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.864519 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.864498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.871243 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.871223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrf9m\" (UniqueName: \"kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:36.941623 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:36.941593 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:37.219999 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:37.219972 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:57:37.221763 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:57:37.221726 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae80571_cf48_4512_8c49_9d95c60b6346.slice/crio-a396eb1c99b033d2d2775c944799f7143edf849153a47147a05cf607b1e9e97d WatchSource:0}: Error finding container a396eb1c99b033d2d2775c944799f7143edf849153a47147a05cf607b1e9e97d: Status 404 returned error can't find the container with id a396eb1c99b033d2d2775c944799f7143edf849153a47147a05cf607b1e9e97d Sep 17 18:57:37.589862 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:37.589783 2559 generic.go:356] "Generic (PLEG): container finished" podID="8cc67759-758d-4876-8ded-d6236e180be9" containerID="4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef" exitCode=2 Sep 17 18:57:37.590231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:37.589863 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerDied","Data":"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef"} Sep 17 18:57:37.591553 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:37.591531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerStarted","Data":"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881"} Sep 17 18:57:37.591654 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:37.591578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerStarted","Data":"a396eb1c99b033d2d2775c944799f7143edf849153a47147a05cf607b1e9e97d"} Sep 17 18:57:39.218838 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:39.218798 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 17 18:57:39.224167 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:39.224137 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 17 18:57:41.121540 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.121513 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:57:41.197338 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197263 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config\") pod \"8cc67759-758d-4876-8ded-d6236e180be9\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " Sep 17 18:57:41.197338 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197300 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location\") pod \"8cc67759-758d-4876-8ded-d6236e180be9\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " Sep 17 18:57:41.197569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197495 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p28gf\" (UniqueName: \"kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf\") pod \"8cc67759-758d-4876-8ded-d6236e180be9\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " Sep 17 18:57:41.197569 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197539 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert\") pod \"8cc67759-758d-4876-8ded-d6236e180be9\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " Sep 17 18:57:41.197679 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197617 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cc67759-758d-4876-8ded-d6236e180be9" (UID: "8cc67759-758d-4876-8ded-d6236e180be9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:57:41.197679 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197643 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls\") pod \"8cc67759-758d-4876-8ded-d6236e180be9\" (UID: \"8cc67759-758d-4876-8ded-d6236e180be9\") " Sep 17 18:57:41.197679 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197643 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cc67759-758d-4876-8ded-d6236e180be9" (UID: "8cc67759-758d-4876-8ded-d6236e180be9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:57:41.197851 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197834 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:57:41.197896 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197857 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cc67759-758d-4876-8ded-d6236e180be9-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:57:41.198014 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.197981 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8cc67759-758d-4876-8ded-d6236e180be9" (UID: "8cc67759-758d-4876-8ded-d6236e180be9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:57:41.199536 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.199513 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf" (OuterVolumeSpecName: "kube-api-access-p28gf") pod "8cc67759-758d-4876-8ded-d6236e180be9" (UID: "8cc67759-758d-4876-8ded-d6236e180be9"). InnerVolumeSpecName "kube-api-access-p28gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:57:41.199617 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.199579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cc67759-758d-4876-8ded-d6236e180be9" (UID: "8cc67759-758d-4876-8ded-d6236e180be9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:57:41.298561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.298534 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p28gf\" (UniqueName: \"kubernetes.io/projected/8cc67759-758d-4876-8ded-d6236e180be9-kube-api-access-p28gf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:57:41.298561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.298559 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cc67759-758d-4876-8ded-d6236e180be9-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:57:41.298561 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.298569 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cc67759-758d-4876-8ded-d6236e180be9-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:57:41.611547 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.611440 2559 generic.go:356] "Generic (PLEG): container finished" podID="8cc67759-758d-4876-8ded-d6236e180be9" containerID="744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6" exitCode=0 Sep 17 18:57:41.611547 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.611487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerDied","Data":"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6"} Sep 17 18:57:41.611547 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.611523 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" Sep 17 18:57:41.611818 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.611581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5" event={"ID":"8cc67759-758d-4876-8ded-d6236e180be9","Type":"ContainerDied","Data":"a5d8f0d07bdf89017c264e8f6f38257926499716422e220ceeef4a4d1c9a1615"} Sep 17 18:57:41.611818 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.611610 2559 scope.go:117] "RemoveContainer" containerID="4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef" Sep 17 18:57:41.613242 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.613207 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerID="99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881" exitCode=0 Sep 17 18:57:41.613411 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.613242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerDied","Data":"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881"} Sep 17 18:57:41.620234 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.620210 2559 scope.go:117] "RemoveContainer" containerID="744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6" Sep 17 18:57:41.627488 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.627469 2559 scope.go:117] "RemoveContainer" containerID="62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19" Sep 17 18:57:41.635915 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.635896 2559 scope.go:117] "RemoveContainer" containerID="4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef" Sep 17 18:57:41.636253 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:57:41.636234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef\": container with ID starting with 4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef not found: ID does not exist" containerID="4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef" Sep 17 18:57:41.636355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.636264 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef"} err="failed to get container status \"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef\": rpc error: code = NotFound desc = could not find container \"4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef\": container with ID starting with 4eec9d610393a6bfedcf83c00c21bebb189680eb67a1e9cdec356d46a62b82ef not found: ID does not exist" Sep 17 18:57:41.636355 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.636285 2559 scope.go:117] "RemoveContainer" containerID="744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6" Sep 17 18:57:41.636582 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:57:41.636562 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6\": container with ID starting with 744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6 not found: ID does not exist" containerID="744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6" Sep 17 18:57:41.636659 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.636589 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6"} err="failed to get container status \"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6\": rpc error: code = NotFound desc = could not find container \"744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6\": container with ID starting with 744148ad00b34b51420d499c4ef99b01f9aab22b1213e4d1c08203b9264fe9b6 not found: ID does not exist" Sep 17 18:57:41.636659 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.636608 2559 scope.go:117] "RemoveContainer" containerID="62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19" Sep 17 18:57:41.636871 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:57:41.636850 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19\": container with ID starting with 62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19 not found: ID does not exist" containerID="62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19" Sep 17 18:57:41.636926 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.636879 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19"} err="failed to get container status \"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19\": rpc error: code = NotFound desc = could not find container \"62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19\": container with ID starting with 62b4be00d53a55fce8c18c0d7832ab371d86d306c9ef9db6c54491ca01689e19 not found: ID does not exist" Sep 17 18:57:41.648641 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.648617 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:57:41.652660 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.652640 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-gf2c5"] Sep 17 18:57:41.992739 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:41.992706 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cc67759-758d-4876-8ded-d6236e180be9" path="/var/lib/kubelet/pods/8cc67759-758d-4876-8ded-d6236e180be9/volumes" Sep 17 18:57:42.620089 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:42.620053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerStarted","Data":"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd"} Sep 17 18:57:42.620590 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:42.620122 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerStarted","Data":"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8"} Sep 17 18:57:42.620590 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:42.620352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:42.644579 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:42.644537 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podStartSLOduration=3.032624894 podStartE2EDuration="6.64452092s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.611896026s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:42.642830715 +0000 UTC m=+1714.631446965" watchObservedRunningTime="2026-09-17 18:57:42.64452092 +0000 UTC m=+1714.633137170" Sep 17 18:57:43.625865 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:43.625833 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:43.627231 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:43.627204 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:57:44.629930 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:44.629890 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:57:45.387645 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:45.387605 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:49.634976 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:49.634942 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:57:49.635558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:49.635529 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:57:55.387027 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:55.386989 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:57:59.636429 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:57:59.636387 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:05.388501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:05.388442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:58:09.635434 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:09.635389 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:12.241924 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:12.241880 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:58:12.242386 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:12.242154 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" containerID="cri-o://c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9" gracePeriod=30 Sep 17 18:58:12.242386 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:12.242258 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kube-rbac-proxy" containerID="cri-o://0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0" gracePeriod=30 Sep 17 18:58:12.755617 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:12.755578 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerID="0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0" exitCode=2 Sep 17 18:58:12.755809 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:12.755665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerDied","Data":"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0"} Sep 17 18:58:15.383209 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:15.383171 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 17 18:58:15.387501 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:15.387474 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 17 18:58:17.188024 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.187999 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:58:17.190950 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.190922 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config\") pod \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " Sep 17 18:58:17.191093 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.190958 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location\") pod \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " Sep 17 18:58:17.191093 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.190993 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert\") pod \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " Sep 17 18:58:17.191093 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191018 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls\") pod \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " Sep 17 18:58:17.191271 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191133 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9hnc\" (UniqueName: \"kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc\") pod \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\" (UID: \"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3\") " Sep 17 18:58:17.191330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191277 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" (UID: "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:58:17.191330 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" (UID: "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:58:17.191415 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" (UID: "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:58:17.191415 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191397 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:58:17.191415 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.191411 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:58:17.193003 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.192981 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" (UID: "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:58:17.193100 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.193082 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc" (OuterVolumeSpecName: "kube-api-access-g9hnc") pod "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" (UID: "fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3"). InnerVolumeSpecName "kube-api-access-g9hnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:58:17.291957 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.291889 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:58:17.291957 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.291917 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:58:17.291957 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.291926 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hnc\" (UniqueName: \"kubernetes.io/projected/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3-kube-api-access-g9hnc\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:58:17.779875 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.779838 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerID="c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9" exitCode=0 Sep 17 18:58:17.780066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.779899 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" Sep 17 18:58:17.780066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.779931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerDied","Data":"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9"} Sep 17 18:58:17.780066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.779991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp" event={"ID":"fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3","Type":"ContainerDied","Data":"81651163d3f53a4077ba3dc53238b0727ec063f88633697e93e07d06454886f3"} Sep 17 18:58:17.780066 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.780019 2559 scope.go:117] "RemoveContainer" containerID="0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0" Sep 17 18:58:17.789229 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.789215 2559 scope.go:117] "RemoveContainer" containerID="c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9" Sep 17 18:58:17.796693 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.796675 2559 scope.go:117] "RemoveContainer" containerID="623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64" Sep 17 18:58:17.802884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.802864 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:58:17.803952 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.803939 2559 scope.go:117] "RemoveContainer" containerID="0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0" Sep 17 18:58:17.804203 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:58:17.804188 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0\": container with ID starting with 0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0 not found: ID does not exist" containerID="0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0" Sep 17 18:58:17.804277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.804210 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0"} err="failed to get container status \"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0\": rpc error: code = NotFound desc = could not find container \"0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0\": container with ID starting with 0d481ace4914a0b6727239f5cb9f7b2f068fa2459e384711eedb33210a2278a0 not found: ID does not exist" Sep 17 18:58:17.804277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.804234 2559 scope.go:117] "RemoveContainer" containerID="c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9" Sep 17 18:58:17.804492 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:58:17.804475 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9\": container with ID starting with c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9 not found: ID does not exist" containerID="c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9" Sep 17 18:58:17.804567 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.804495 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9"} err="failed to get container status \"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9\": rpc error: code = NotFound desc = could not find container \"c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9\": container with ID starting with c82c91a7fc19ab0e50db627486020b7ab60b36afa3e93417b18746e0c8866ed9 not found: ID does not exist" Sep 17 18:58:17.804567 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.804508 2559 scope.go:117] "RemoveContainer" containerID="623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64" Sep 17 18:58:17.804779 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:58:17.804761 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64\": container with ID starting with 623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64 not found: ID does not exist" containerID="623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64" Sep 17 18:58:17.804826 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.804784 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64"} err="failed to get container status \"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64\": rpc error: code = NotFound desc = could not find container \"623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64\": container with ID starting with 623afa4f0c9aa2c117286c8400357484581bde5bb53a962faca6c1499edc9e64 not found: ID does not exist" Sep 17 18:58:17.808696 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.808677 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7df954dcd9-v5vwp"] Sep 17 18:58:17.992460 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:17.992415 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" path="/var/lib/kubelet/pods/fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3/volumes" Sep 17 18:58:19.636400 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:19.636360 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:29.635758 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:29.635715 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:39.635649 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:39.635608 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:49.635537 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:49.635500 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:58:59.636179 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:58:59.636105 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:59:06.774414 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:06.774381 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:59:06.774849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:06.774656 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" containerID="cri-o://4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8" gracePeriod=30 Sep 17 18:59:06.774849 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:06.774682 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kube-rbac-proxy" containerID="cri-o://e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd" gracePeriod=30 Sep 17 18:59:06.999383 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:06.999349 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerID="e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd" exitCode=2 Sep 17 18:59:06.999541 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:06.999421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerDied","Data":"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd"} Sep 17 18:59:09.630295 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:09.630255 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 17 18:59:09.635608 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:09.635582 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 17 18:59:10.031526 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:10.031497 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:59:10.032353 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:10.032330 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 18:59:10.042606 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:10.042581 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:59:10.043358 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:10.043336 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 18:59:11.408658 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.408631 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:59:11.512591 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512528 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location\") pod \"9ae80571-cf48-4512-8c49-9d95c60b6346\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " Sep 17 18:59:11.512591 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512576 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls\") pod \"9ae80571-cf48-4512-8c49-9d95c60b6346\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " Sep 17 18:59:11.512813 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512614 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config\") pod \"9ae80571-cf48-4512-8c49-9d95c60b6346\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " Sep 17 18:59:11.512813 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512631 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert\") pod \"9ae80571-cf48-4512-8c49-9d95c60b6346\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " Sep 17 18:59:11.512813 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512650 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrf9m\" (UniqueName: \"kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m\") pod \"9ae80571-cf48-4512-8c49-9d95c60b6346\" (UID: \"9ae80571-cf48-4512-8c49-9d95c60b6346\") " Sep 17 18:59:11.512975 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.512876 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ae80571-cf48-4512-8c49-9d95c60b6346" (UID: "9ae80571-cf48-4512-8c49-9d95c60b6346"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:59:11.513037 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.513008 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ae80571-cf48-4512-8c49-9d95c60b6346" (UID: "9ae80571-cf48-4512-8c49-9d95c60b6346"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:59:11.513083 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.513018 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ae80571-cf48-4512-8c49-9d95c60b6346" (UID: "9ae80571-cf48-4512-8c49-9d95c60b6346"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:59:11.514901 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.514874 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ae80571-cf48-4512-8c49-9d95c60b6346" (UID: "9ae80571-cf48-4512-8c49-9d95c60b6346"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:59:11.514901 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.514875 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m" (OuterVolumeSpecName: "kube-api-access-nrf9m") pod "9ae80571-cf48-4512-8c49-9d95c60b6346" (UID: "9ae80571-cf48-4512-8c49-9d95c60b6346"). InnerVolumeSpecName "kube-api-access-nrf9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:59:11.613407 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.613379 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae80571-cf48-4512-8c49-9d95c60b6346-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:59:11.613407 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.613402 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae80571-cf48-4512-8c49-9d95c60b6346-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:59:11.613571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.613414 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:59:11.613571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.613423 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae80571-cf48-4512-8c49-9d95c60b6346-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:59:11.613571 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.613431 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrf9m\" (UniqueName: \"kubernetes.io/projected/9ae80571-cf48-4512-8c49-9d95c60b6346-kube-api-access-nrf9m\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 18:59:11.978209 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978169 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 18:59:11.978492 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978479 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="storage-initializer" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978493 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="storage-initializer" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978503 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978509 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978517 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="storage-initializer" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978522 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="storage-initializer" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978531 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kube-rbac-proxy" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978536 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kube-rbac-proxy" Sep 17 18:59:11.978544 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978545 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978552 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978562 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978567 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978577 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978583 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978590 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="storage-initializer" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978595 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="storage-initializer" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978602 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978608 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978655 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978664 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kube-rbac-proxy" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978670 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978676 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978681 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1ef929-5afc-4b9a-a2b8-f6b256e1b1c3" containerName="kserve-container" Sep 17 18:59:11.978785 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.978686 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc67759-758d-4876-8ded-d6236e180be9" containerName="kube-rbac-proxy" Sep 17 18:59:11.981944 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.981927 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:11.984724 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.984695 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Sep 17 18:59:11.984817 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.984725 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 17 18:59:11.993508 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:11.993487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 18:59:12.016304 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.016277 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 18:59:12.019618 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.019599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.022416 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022379 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ae80571-cf48-4512-8c49-9d95c60b6346" containerID="4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8" exitCode=0 Sep 17 18:59:12.022564 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" Sep 17 18:59:12.022564 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerDied","Data":"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8"} Sep 17 18:59:12.022691 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf" event={"ID":"9ae80571-cf48-4512-8c49-9d95c60b6346","Type":"ContainerDied","Data":"a396eb1c99b033d2d2775c944799f7143edf849153a47147a05cf607b1e9e97d"} Sep 17 18:59:12.022691 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022599 2559 scope.go:117] "RemoveContainer" containerID="e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd" Sep 17 18:59:12.022994 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.022911 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 17 18:59:12.031870 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.031848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 18:59:12.032725 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.032708 2559 scope.go:117] "RemoveContainer" containerID="4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8" Sep 17 18:59:12.041423 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.041399 2559 scope.go:117] "RemoveContainer" containerID="99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881" Sep 17 18:59:12.048790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.048768 2559 scope.go:117] "RemoveContainer" containerID="e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd" Sep 17 18:59:12.049051 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:59:12.049032 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd\": container with ID starting with e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd not found: ID does not exist" containerID="e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd" Sep 17 18:59:12.049116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.049058 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd"} err="failed to get container status \"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd\": rpc error: code = NotFound desc = could not find container \"e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd\": container with ID starting with e377a3009af8939fcd0b6eadcdfb984f2b5a9b2d78d04dbde586a60d7bbca8fd not found: ID does not exist" Sep 17 18:59:12.049116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.049076 2559 scope.go:117] "RemoveContainer" containerID="4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8" Sep 17 18:59:12.049383 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:59:12.049366 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8\": container with ID starting with 4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8 not found: ID does not exist" containerID="4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8" Sep 17 18:59:12.049539 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.049386 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8"} err="failed to get container status \"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8\": rpc error: code = NotFound desc = could not find container \"4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8\": container with ID starting with 4c793b6cd3d41a272711080e8715a66488914a28c165b2b20a2dd6fd989f55e8 not found: ID does not exist" Sep 17 18:59:12.049539 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.049399 2559 scope.go:117] "RemoveContainer" containerID="99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881" Sep 17 18:59:12.049671 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:59:12.049650 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881\": container with ID starting with 99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881 not found: ID does not exist" containerID="99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881" Sep 17 18:59:12.049730 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.049675 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881"} err="failed to get container status \"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881\": rpc error: code = NotFound desc = could not find container \"99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881\": container with ID starting with 99ac13c93e75a89fe7536eae9211f584c12e91e59918bd572c4c1387e1b0c881 not found: ID does not exist" Sep 17 18:59:12.073023 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.072991 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:59:12.082054 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.082029 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-bw6cf"] Sep 17 18:59:12.116638 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.116757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.116757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.116757 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.116866 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhlg\" (UniqueName: \"kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.116866 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsnz\" (UniqueName: \"kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.116866 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.116954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.116954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.116954 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.116912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.217721 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217691 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhlg\" (UniqueName: \"kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsnz\" (UniqueName: \"kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.217884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.218218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.218218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217941 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.218218 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.217966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.218218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:59:12.218079 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 17 18:59:12.218218 ip-10-0-132-9 kubenswrapper[2559]: E0917 18:59:12.218145 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls podName:8780c564-5656-4109-983e-50701cfcd06e nodeName:}" failed. No retries permitted until 2026-09-17 18:59:12.718125874 +0000 UTC m=+1804.706742104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls") pod "isvc-canary-rollback-predictor-7679bbb844-xfbcv" (UID: "8780c564-5656-4109-983e-50701cfcd06e") : secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 17 18:59:12.218545 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.218545 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.218656 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218546 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.218715 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.218890 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.218957 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.218893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.220277 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.220254 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.226411 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.226392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsnz\" (UniqueName: \"kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.227097 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.227072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhlg\" (UniqueName: \"kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.293116 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.293051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:12.564784 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.564756 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 18:59:12.566984 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:59:12.566954 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd1575c_26b0_4982_bf04_49828318b800.slice/crio-b8a4e161cc3f668d7885e496be0c93ba5d10f4603989990ba9a2df8cd7f43417 WatchSource:0}: Error finding container b8a4e161cc3f668d7885e496be0c93ba5d10f4603989990ba9a2df8cd7f43417: Status 404 returned error can't find the container with id b8a4e161cc3f668d7885e496be0c93ba5d10f4603989990ba9a2df8cd7f43417 Sep 17 18:59:12.721704 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.721669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.723973 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.723946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-xfbcv\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:12.931891 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:12.931856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:13.030012 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:13.029974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerStarted","Data":"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c"} Sep 17 18:59:13.030193 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:13.030051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerStarted","Data":"b8a4e161cc3f668d7885e496be0c93ba5d10f4603989990ba9a2df8cd7f43417"} Sep 17 18:59:13.204120 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:13.204086 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 18:59:13.205548 ip-10-0-132-9 kubenswrapper[2559]: W0917 18:59:13.205513 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8780c564_5656_4109_983e_50701cfcd06e.slice/crio-6209f55bdc39a323da2b2cf9493345d211141bd74d212629f33c37e2ed68e5cb WatchSource:0}: Error finding container 6209f55bdc39a323da2b2cf9493345d211141bd74d212629f33c37e2ed68e5cb: Status 404 returned error can't find the container with id 6209f55bdc39a323da2b2cf9493345d211141bd74d212629f33c37e2ed68e5cb Sep 17 18:59:13.991925 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:13.991888 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae80571-cf48-4512-8c49-9d95c60b6346" path="/var/lib/kubelet/pods/9ae80571-cf48-4512-8c49-9d95c60b6346/volumes" Sep 17 18:59:14.034764 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:14.034733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerStarted","Data":"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc"} Sep 17 18:59:14.034871 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:14.034799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerStarted","Data":"6209f55bdc39a323da2b2cf9493345d211141bd74d212629f33c37e2ed68e5cb"} Sep 17 18:59:17.047884 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:17.047854 2559 generic.go:356] "Generic (PLEG): container finished" podID="3bd1575c-26b0-4982-bf04-49828318b800" containerID="7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c" exitCode=0 Sep 17 18:59:17.048250 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:17.047929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerDied","Data":"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c"} Sep 17 18:59:17.049576 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:17.049555 2559 generic.go:356] "Generic (PLEG): container finished" podID="8780c564-5656-4109-983e-50701cfcd06e" containerID="7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc" exitCode=0 Sep 17 18:59:17.049654 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:17.049591 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerDied","Data":"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc"} Sep 17 18:59:18.057558 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.057512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerStarted","Data":"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1"} Sep 17 18:59:18.058031 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.057598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerStarted","Data":"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052"} Sep 17 18:59:18.058031 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.057842 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:18.059604 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.059578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerStarted","Data":"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc"} Sep 17 18:59:18.059724 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.059627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerStarted","Data":"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f"} Sep 17 18:59:18.059814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.059799 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:18.059857 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.059838 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:18.060979 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.060956 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:18.082640 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.082592 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podStartSLOduration=3.291067009 podStartE2EDuration="7.082579883s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791512874s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:18.081061978 +0000 UTC m=+1810.069678227" watchObservedRunningTime="2026-09-17 18:59:18.082579883 +0000 UTC m=+1810.071196186" Sep 17 18:59:18.106922 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:18.106874 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podStartSLOduration=3.413523489 podStartE2EDuration="7.106862272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.693338783s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:18.105223103 +0000 UTC m=+1810.093839351" watchObservedRunningTime="2026-09-17 18:59:18.106862272 +0000 UTC m=+1810.095478521" Sep 17 18:59:19.064296 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:19.064258 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:19.064771 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:19.064486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:19.065545 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:19.065521 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 18:59:20.069045 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:20.069001 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 18:59:24.068814 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:24.068782 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 18:59:24.069356 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:24.069329 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:25.074011 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:25.073977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 18:59:25.074607 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:25.074582 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 18:59:34.069790 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:34.069696 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:35.074911 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:35.074864 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 18:59:44.069631 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:44.069593 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:45.074768 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:45.074727 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 18:59:54.069538 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:54.069503 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 18:59:55.074531 ip-10-0-132-9 kubenswrapper[2559]: I0917 18:59:55.074488 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 19:00:04.070210 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:04.070169 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 19:00:05.074513 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:05.074472 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 19:00:14.069890 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:14.069852 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 19:00:15.075024 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:15.074982 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 19:00:24.069643 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:24.069553 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 19:00:25.075274 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:25.075233 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 19:00:34.070816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:34.070735 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 19:00:35.075593 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:35.075560 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 19:00:38.517121 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.517092 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:00:38.520455 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.520430 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.524728 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:38.524700 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.524728 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:38.524711 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-tensorflow-runtime-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\"" type="*v1.Secret" Sep 17 19:00:38.525444 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:38.525412 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap" Sep 17 19:00:38.543845 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.543819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:00:38.640395 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.640357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.640395 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.640394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.640685 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.640498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.640685 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.640525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8nx\" (UniqueName: \"kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.640685 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.640560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741358 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741511 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741511 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741511 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741511 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8nx\" (UniqueName: \"kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.741827 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.741803 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.742093 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.742075 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:38.750063 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:38.750043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8nx\" (UniqueName: \"kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:39.404887 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.404856 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\"" Sep 17 19:00:39.413803 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.413777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:39.538874 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.538842 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:00:39.542414 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.542384 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-4pgld\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:39.730289 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.730215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:39.999435 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:39.999404 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:00:40.000541 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:00:40.000516 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba347d3_4a52_4308_b7ad_dd847ce53b5c.slice/crio-2c9f927462933d62a4e81e1ced50020f4942058a3d7ba1942ce313848831041b WatchSource:0}: Error finding container 2c9f927462933d62a4e81e1ced50020f4942058a3d7ba1942ce313848831041b: Status 404 returned error can't find the container with id 2c9f927462933d62a4e81e1ced50020f4942058a3d7ba1942ce313848831041b Sep 17 19:00:40.413675 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:40.413639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerStarted","Data":"85acf3b75bc9e00e7b0df0691d4530e6690c34f1804e9adedbd44f45009c074f"} Sep 17 19:00:40.413855 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:40.413710 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerStarted","Data":"2c9f927462933d62a4e81e1ced50020f4942058a3d7ba1942ce313848831041b"} Sep 17 19:00:41.977618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:41.977576 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 19:00:41.978042 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:41.977809 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" containerID="cri-o://cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052" gracePeriod=30 Sep 17 19:00:41.978042 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:41.977845 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kube-rbac-proxy" containerID="cri-o://639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1" gracePeriod=30 Sep 17 19:00:42.426383 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:42.426353 2559 generic.go:356] "Generic (PLEG): container finished" podID="3bd1575c-26b0-4982-bf04-49828318b800" containerID="639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1" exitCode=2 Sep 17 19:00:42.426571 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:42.426422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerDied","Data":"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1"} Sep 17 19:00:45.069463 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:45.069356 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 17 19:00:45.074460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:45.074412 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 17 19:00:45.441771 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:45.441733 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerID="85acf3b75bc9e00e7b0df0691d4530e6690c34f1804e9adedbd44f45009c074f" exitCode=0 Sep 17 19:00:45.441938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:45.441806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerDied","Data":"85acf3b75bc9e00e7b0df0691d4530e6690c34f1804e9adedbd44f45009c074f"} Sep 17 19:00:45.443013 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:45.442993 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 19:00:47.095141 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.095084 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 19:00:47.215813 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.215776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location\") pod \"3bd1575c-26b0-4982-bf04-49828318b800\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " Sep 17 19:00:47.216001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.215830 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls\") pod \"3bd1575c-26b0-4982-bf04-49828318b800\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " Sep 17 19:00:47.216001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.215875 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config\") pod \"3bd1575c-26b0-4982-bf04-49828318b800\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " Sep 17 19:00:47.216001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.215907 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spsnz\" (UniqueName: \"kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz\") pod \"3bd1575c-26b0-4982-bf04-49828318b800\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " Sep 17 19:00:47.216001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.215956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert\") pod \"3bd1575c-26b0-4982-bf04-49828318b800\" (UID: \"3bd1575c-26b0-4982-bf04-49828318b800\") " Sep 17 19:00:47.216320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.216288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bd1575c-26b0-4982-bf04-49828318b800" (UID: "3bd1575c-26b0-4982-bf04-49828318b800"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:00:47.216489 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.216438 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bd1575c-26b0-4982-bf04-49828318b800" (UID: "3bd1575c-26b0-4982-bf04-49828318b800"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:00:47.216489 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.216473 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3bd1575c-26b0-4982-bf04-49828318b800" (UID: "3bd1575c-26b0-4982-bf04-49828318b800"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:00:47.218371 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.218345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz" (OuterVolumeSpecName: "kube-api-access-spsnz") pod "3bd1575c-26b0-4982-bf04-49828318b800" (UID: "3bd1575c-26b0-4982-bf04-49828318b800"). InnerVolumeSpecName "kube-api-access-spsnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:00:47.218513 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.218442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bd1575c-26b0-4982-bf04-49828318b800" (UID: "3bd1575c-26b0-4982-bf04-49828318b800"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:00:47.316954 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.316922 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd1575c-26b0-4982-bf04-49828318b800-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:00:47.316954 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.316950 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd1575c-26b0-4982-bf04-49828318b800-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:00:47.316954 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.316960 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:00:47.317187 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.316969 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsnz\" (UniqueName: \"kubernetes.io/projected/3bd1575c-26b0-4982-bf04-49828318b800-kube-api-access-spsnz\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:00:47.317187 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.316982 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd1575c-26b0-4982-bf04-49828318b800-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:00:47.454347 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.454312 2559 generic.go:356] "Generic (PLEG): container finished" podID="3bd1575c-26b0-4982-bf04-49828318b800" containerID="cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052" exitCode=0 Sep 17 19:00:47.454555 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.454379 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" Sep 17 19:00:47.454555 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.454399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerDied","Data":"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052"} Sep 17 19:00:47.454555 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.454493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq" event={"ID":"3bd1575c-26b0-4982-bf04-49828318b800","Type":"ContainerDied","Data":"b8a4e161cc3f668d7885e496be0c93ba5d10f4603989990ba9a2df8cd7f43417"} Sep 17 19:00:47.454555 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.454533 2559 scope.go:117] "RemoveContainer" containerID="639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1" Sep 17 19:00:47.465135 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.465112 2559 scope.go:117] "RemoveContainer" containerID="cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052" Sep 17 19:00:47.474146 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.474128 2559 scope.go:117] "RemoveContainer" containerID="7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c" Sep 17 19:00:47.480504 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.480483 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 19:00:47.483478 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.483435 2559 scope.go:117] "RemoveContainer" containerID="639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1" Sep 17 19:00:47.483755 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:47.483737 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1\": container with ID starting with 639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1 not found: ID does not exist" containerID="639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1" Sep 17 19:00:47.483843 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.483762 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1"} err="failed to get container status \"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1\": rpc error: code = NotFound desc = could not find container \"639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1\": container with ID starting with 639363f5fa963c8c649194bec3128485d5ac1614ceb201aca5b75df083db28b1 not found: ID does not exist" Sep 17 19:00:47.483843 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.483777 2559 scope.go:117] "RemoveContainer" containerID="cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052" Sep 17 19:00:47.484040 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:47.484022 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052\": container with ID starting with cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052 not found: ID does not exist" containerID="cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052" Sep 17 19:00:47.484099 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.484045 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052"} err="failed to get container status \"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052\": rpc error: code = NotFound desc = could not find container \"cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052\": container with ID starting with cf6ccd7452414713886ea084d04c1e697e48026b8e8472fd5eb771708457e052 not found: ID does not exist" Sep 17 19:00:47.484099 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.484060 2559 scope.go:117] "RemoveContainer" containerID="7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c" Sep 17 19:00:47.484391 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:00:47.484371 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c\": container with ID starting with 7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c not found: ID does not exist" containerID="7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c" Sep 17 19:00:47.484491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.484399 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c"} err="failed to get container status \"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c\": rpc error: code = NotFound desc = could not find container \"7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c\": container with ID starting with 7ddac4894c6b547a997864d569a2b4270a10649d77f8c61849e73475d4d78a2c not found: ID does not exist" Sep 17 19:00:47.484491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.484381 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-mwqhq"] Sep 17 19:00:47.993764 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:47.993730 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd1575c-26b0-4982-bf04-49828318b800" path="/var/lib/kubelet/pods/3bd1575c-26b0-4982-bf04-49828318b800/volumes" Sep 17 19:00:49.467462 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:49.467425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerStarted","Data":"b75336599e43c0baf5a4a252d771b1fb2a1a766fa475103c2691977a98e2714e"} Sep 17 19:00:49.467925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:49.467519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerStarted","Data":"d46eb6d96dc9113c8aa0c74f25cc2c5617cc5b5948b882e18d4507280254634e"} Sep 17 19:00:49.467925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:49.467833 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:49.489751 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:49.489666 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podStartSLOduration=3.2069856740000002 podStartE2EDuration="11.489651262s" totalImagesPullingTime="3.56049311s" totalInitContainerRuntime="4.722172478s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:49.488537694 +0000 UTC m=+1901.477153943" watchObservedRunningTime="2026-09-17 19:00:49.489651262 +0000 UTC m=+1901.478267511" Sep 17 19:00:50.472714 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:50.472687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:50.474099 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:50.474070 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 17 19:00:51.477973 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:51.477934 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 17 19:00:56.482270 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:56.482240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:00:56.482788 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:00:56.482760 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 17 19:01:06.483297 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:06.483265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:01:18.261248 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:18.261206 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:01:18.261699 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:18.261439 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" containerID="cri-o://d46eb6d96dc9113c8aa0c74f25cc2c5617cc5b5948b882e18d4507280254634e" gracePeriod=30 Sep 17 19:01:18.261699 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:18.261576 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" containerID="cri-o://b75336599e43c0baf5a4a252d771b1fb2a1a766fa475103c2691977a98e2714e" gracePeriod=30 Sep 17 19:01:18.602944 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:18.602859 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerID="b75336599e43c0baf5a4a252d771b1fb2a1a766fa475103c2691977a98e2714e" exitCode=2 Sep 17 19:01:18.602944 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:18.602932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerDied","Data":"b75336599e43c0baf5a4a252d771b1fb2a1a766fa475103c2691977a98e2714e"} Sep 17 19:01:21.479137 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:21.479093 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:26.478890 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:26.478845 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:31.479173 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:31.479133 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:31.479593 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:31.479270 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:01:36.478257 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:36.478210 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:41.478903 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:41.478864 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:46.478614 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:46.478567 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 17 19:01:48.738580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:48.738549 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerID="d46eb6d96dc9113c8aa0c74f25cc2c5617cc5b5948b882e18d4507280254634e" exitCode=137 Sep 17 19:01:48.738951 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:48.738618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerDied","Data":"d46eb6d96dc9113c8aa0c74f25cc2c5617cc5b5948b882e18d4507280254634e"} Sep 17 19:01:48.909925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:48.909897 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:01:49.011352 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011269 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls\") pod \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " Sep 17 19:01:49.011352 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011324 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config\") pod \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " Sep 17 19:01:49.011352 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl8nx\" (UniqueName: \"kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx\") pod \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " Sep 17 19:01:49.011595 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011468 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert\") pod \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " Sep 17 19:01:49.011595 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011576 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location\") pod \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\" (UID: \"9ba347d3-4a52-4308-b7ad-dd847ce53b5c\") " Sep 17 19:01:49.011755 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011732 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ba347d3-4a52-4308-b7ad-dd847ce53b5c" (UID: "9ba347d3-4a52-4308-b7ad-dd847ce53b5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:01:49.011845 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011827 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:01:49.011897 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.011835 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ba347d3-4a52-4308-b7ad-dd847ce53b5c" (UID: "9ba347d3-4a52-4308-b7ad-dd847ce53b5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:01:49.013531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.013501 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ba347d3-4a52-4308-b7ad-dd847ce53b5c" (UID: "9ba347d3-4a52-4308-b7ad-dd847ce53b5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:01:49.013531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.013494 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx" (OuterVolumeSpecName: "kube-api-access-tl8nx") pod "9ba347d3-4a52-4308-b7ad-dd847ce53b5c" (UID: "9ba347d3-4a52-4308-b7ad-dd847ce53b5c"). InnerVolumeSpecName "kube-api-access-tl8nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:01:49.021615 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.021589 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ba347d3-4a52-4308-b7ad-dd847ce53b5c" (UID: "9ba347d3-4a52-4308-b7ad-dd847ce53b5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:01:49.112367 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.112342 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:01:49.112367 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.112366 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:01:49.112543 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.112379 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8nx\" (UniqueName: \"kubernetes.io/projected/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-kube-api-access-tl8nx\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:01:49.112543 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.112392 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ba347d3-4a52-4308-b7ad-dd847ce53b5c-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:01:49.744354 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.744313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" event={"ID":"9ba347d3-4a52-4308-b7ad-dd847ce53b5c","Type":"ContainerDied","Data":"2c9f927462933d62a4e81e1ced50020f4942058a3d7ba1942ce313848831041b"} Sep 17 19:01:49.744354 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.744349 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld" Sep 17 19:01:49.744923 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.744383 2559 scope.go:117] "RemoveContainer" containerID="b75336599e43c0baf5a4a252d771b1fb2a1a766fa475103c2691977a98e2714e" Sep 17 19:01:49.753199 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.753182 2559 scope.go:117] "RemoveContainer" containerID="d46eb6d96dc9113c8aa0c74f25cc2c5617cc5b5948b882e18d4507280254634e" Sep 17 19:01:49.760654 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.760633 2559 scope.go:117] "RemoveContainer" containerID="85acf3b75bc9e00e7b0df0691d4530e6690c34f1804e9adedbd44f45009c074f" Sep 17 19:01:49.768276 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.768256 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:01:49.771545 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.771523 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-4pgld"] Sep 17 19:01:49.992736 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:01:49.992709 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" path="/var/lib/kubelet/pods/9ba347d3-4a52-4308-b7ad-dd847ce53b5c/volumes" Sep 17 19:02:22.128973 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:22.128935 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 19:02:22.129437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:22.129185 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" containerID="cri-o://6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f" gracePeriod=30 Sep 17 19:02:22.129437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:22.129208 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kube-rbac-proxy" containerID="cri-o://920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc" gracePeriod=30 Sep 17 19:02:22.885222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:22.885180 2559 generic.go:356] "Generic (PLEG): container finished" podID="8780c564-5656-4109-983e-50701cfcd06e" containerID="920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc" exitCode=2 Sep 17 19:02:22.885417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:22.885246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerDied","Data":"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc"} Sep 17 19:02:24.064886 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:24.064841 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused" Sep 17 19:02:24.069294 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:24.069264 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 17 19:02:27.082004 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.081978 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 19:02:27.098898 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.098873 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert\") pod \"8780c564-5656-4109-983e-50701cfcd06e\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " Sep 17 19:02:27.099090 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.098914 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hhlg\" (UniqueName: \"kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg\") pod \"8780c564-5656-4109-983e-50701cfcd06e\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " Sep 17 19:02:27.099090 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.098935 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") pod \"8780c564-5656-4109-983e-50701cfcd06e\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " Sep 17 19:02:27.099090 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.098965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config\") pod \"8780c564-5656-4109-983e-50701cfcd06e\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " Sep 17 19:02:27.099090 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.098989 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location\") pod \"8780c564-5656-4109-983e-50701cfcd06e\" (UID: \"8780c564-5656-4109-983e-50701cfcd06e\") " Sep 17 19:02:27.099325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.099265 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8780c564-5656-4109-983e-50701cfcd06e" (UID: "8780c564-5656-4109-983e-50701cfcd06e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:02:27.099383 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.099366 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8780c564-5656-4109-983e-50701cfcd06e" (UID: "8780c564-5656-4109-983e-50701cfcd06e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:02:27.099460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.099403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8780c564-5656-4109-983e-50701cfcd06e" (UID: "8780c564-5656-4109-983e-50701cfcd06e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:02:27.100936 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.100913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8780c564-5656-4109-983e-50701cfcd06e" (UID: "8780c564-5656-4109-983e-50701cfcd06e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:02:27.101420 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.101398 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg" (OuterVolumeSpecName: "kube-api-access-7hhlg") pod "8780c564-5656-4109-983e-50701cfcd06e" (UID: "8780c564-5656-4109-983e-50701cfcd06e"). InnerVolumeSpecName "kube-api-access-7hhlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:02:27.199548 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.199509 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:02:27.199548 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.199539 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhlg\" (UniqueName: \"kubernetes.io/projected/8780c564-5656-4109-983e-50701cfcd06e-kube-api-access-7hhlg\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:02:27.199548 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.199549 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8780c564-5656-4109-983e-50701cfcd06e-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:02:27.199548 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.199558 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8780c564-5656-4109-983e-50701cfcd06e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:02:27.199817 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.199567 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8780c564-5656-4109-983e-50701cfcd06e-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:02:27.325958 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.325922 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:02:27.326243 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326231 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kube-rbac-proxy" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326245 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kube-rbac-proxy" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326255 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kube-rbac-proxy" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326260 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kube-rbac-proxy" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326270 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326275 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" Sep 17 19:02:27.326288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326283 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326292 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326300 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326305 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326321 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326327 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326335 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326340 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326351 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="storage-initializer" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326356 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326363 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326413 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kube-rbac-proxy" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326419 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd1575c-26b0-4982-bf04-49828318b800" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326425 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kube-rbac-proxy" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326430 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kserve-container" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326436 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba347d3-4a52-4308-b7ad-dd847ce53b5c" containerName="kube-rbac-proxy" Sep 17 19:02:27.326498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.326441 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8780c564-5656-4109-983e-50701cfcd06e" containerName="kserve-container" Sep 17 19:02:27.329741 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.329726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.332173 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.332139 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\"" Sep 17 19:02:27.332550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.332523 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 17 19:02:27.339860 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.339837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:02:27.375198 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.375120 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:02:27.378543 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.378524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.385187 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.385162 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 17 19:02:27.400745 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.400708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.400994 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.400974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.401118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.401244 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.401343 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.401462 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.401588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.401685 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2ch\" (UniqueName: \"kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.401791 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnvm\" (UniqueName: \"kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.401913 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.401900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.402092 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.402077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:02:27.503139 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnvm\" (UniqueName: \"kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.503291 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.503291 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.503291 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.503291 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.503554 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.503554 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.503662 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.503662 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.503662 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.503620 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2ch\" (UniqueName: \"kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.504038 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.504149 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.504223 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.504223 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.504302 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.504413 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.504386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.505996 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.505968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.506241 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.506222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.511163 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.511143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2ch\" (UniqueName: \"kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.511251 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.511193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnvm\" (UniqueName: \"kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm\") pod \"isvc-canary-stop-predictor-7968cbc88d-wlq55\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.640151 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.640115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:27.688768 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.688556 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:27.907762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.907679 2559 generic.go:356] "Generic (PLEG): container finished" podID="8780c564-5656-4109-983e-50701cfcd06e" containerID="6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f" exitCode=0 Sep 17 19:02:27.907762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.907747 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" Sep 17 19:02:27.907941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.907764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerDied","Data":"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f"} Sep 17 19:02:27.907941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.907825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv" event={"ID":"8780c564-5656-4109-983e-50701cfcd06e","Type":"ContainerDied","Data":"6209f55bdc39a323da2b2cf9493345d211141bd74d212629f33c37e2ed68e5cb"} Sep 17 19:02:27.907941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.907853 2559 scope.go:117] "RemoveContainer" containerID="920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc" Sep 17 19:02:27.918066 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.918048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:02:27.918409 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.918391 2559 scope.go:117] "RemoveContainer" containerID="6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f" Sep 17 19:02:27.919985 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:02:27.919961 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd682c13_6191_42c7_bb8b_8cfd1724f652.slice/crio-12cff204c59e97044fc334f05adeeefdb60003df2bab869f62a88bab06643b20 WatchSource:0}: Error finding container 12cff204c59e97044fc334f05adeeefdb60003df2bab869f62a88bab06643b20: Status 404 returned error can't find the container with id 12cff204c59e97044fc334f05adeeefdb60003df2bab869f62a88bab06643b20 Sep 17 19:02:27.929824 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.929800 2559 scope.go:117] "RemoveContainer" containerID="7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc" Sep 17 19:02:27.934288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.934199 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 19:02:27.936933 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.936909 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-xfbcv"] Sep 17 19:02:27.939062 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939044 2559 scope.go:117] "RemoveContainer" containerID="920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc" Sep 17 19:02:27.939431 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:02:27.939337 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc\": container with ID starting with 920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc not found: ID does not exist" containerID="920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc" Sep 17 19:02:27.939431 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939361 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc"} err="failed to get container status \"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc\": rpc error: code = NotFound desc = could not find container \"920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc\": container with ID starting with 920b9e66f5c0363217db69c8550d7cfb1c83e157e2eafdf5efecc4aefb83f0cc not found: ID does not exist" Sep 17 19:02:27.939431 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939379 2559 scope.go:117] "RemoveContainer" containerID="6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f" Sep 17 19:02:27.939686 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:02:27.939662 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f\": container with ID starting with 6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f not found: ID does not exist" containerID="6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f" Sep 17 19:02:27.939752 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939693 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f"} err="failed to get container status \"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f\": rpc error: code = NotFound desc = could not find container \"6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f\": container with ID starting with 6fa5669378980b7d28f5485445768ad099663b93ce2d4f11de66edc212bd266f not found: ID does not exist" Sep 17 19:02:27.939752 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939716 2559 scope.go:117] "RemoveContainer" containerID="7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc" Sep 17 19:02:27.939986 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:02:27.939967 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc\": container with ID starting with 7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc not found: ID does not exist" containerID="7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc" Sep 17 19:02:27.940044 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.939990 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc"} err="failed to get container status \"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc\": rpc error: code = NotFound desc = could not find container \"7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc\": container with ID starting with 7d0c6546ad928734f772e6e50383e29207b2e1d486803eb6f65066e6ad9c60cc not found: ID does not exist" Sep 17 19:02:27.967279 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.967250 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:02:27.971216 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:02:27.971192 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0330c_38d3_4146_9095_22786ba33155.slice/crio-ff18c14af55106196beb448c8219c0b4c06e5b59938f87af4882abecffedcd56 WatchSource:0}: Error finding container ff18c14af55106196beb448c8219c0b4c06e5b59938f87af4882abecffedcd56: Status 404 returned error can't find the container with id ff18c14af55106196beb448c8219c0b4c06e5b59938f87af4882abecffedcd56 Sep 17 19:02:27.995955 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:27.995931 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8780c564-5656-4109-983e-50701cfcd06e" path="/var/lib/kubelet/pods/8780c564-5656-4109-983e-50701cfcd06e/volumes" Sep 17 19:02:28.914908 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:28.914866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerStarted","Data":"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15"} Sep 17 19:02:28.915336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:28.914938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerStarted","Data":"ff18c14af55106196beb448c8219c0b4c06e5b59938f87af4882abecffedcd56"} Sep 17 19:02:28.916379 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:28.916352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerStarted","Data":"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce"} Sep 17 19:02:28.916490 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:28.916402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerStarted","Data":"12cff204c59e97044fc334f05adeeefdb60003df2bab869f62a88bab06643b20"} Sep 17 19:02:32.936718 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:32.936680 2559 generic.go:356] "Generic (PLEG): container finished" podID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerID="4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce" exitCode=0 Sep 17 19:02:32.937143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:32.936748 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerDied","Data":"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce"} Sep 17 19:02:32.938485 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:32.938462 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ef0330c-38d3-4146-9095-22786ba33155" containerID="f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15" exitCode=0 Sep 17 19:02:32.938565 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:32.938521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerDied","Data":"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15"} Sep 17 19:02:33.944675 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.944636 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerStarted","Data":"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361"} Sep 17 19:02:33.945109 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.944700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerStarted","Data":"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744"} Sep 17 19:02:33.945109 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.944905 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:33.946793 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.946762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerStarted","Data":"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f"} Sep 17 19:02:33.946930 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.946818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerStarted","Data":"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3"} Sep 17 19:02:33.947088 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.947069 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:33.947160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.947110 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:33.948816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.948780 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:02:33.974373 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:33.974324 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podStartSLOduration=3.009924855 podStartE2EDuration="6.974311297s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.964386442s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:33.972903806 +0000 UTC m=+2005.961520056" watchObservedRunningTime="2026-09-17 19:02:33.974311297 +0000 UTC m=+2005.962927546" Sep 17 19:02:34.005322 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:34.005279 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podStartSLOduration=2.80826788 podStartE2EDuration="7.005266643s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.196998763s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:34.004049787 +0000 UTC m=+2005.992666047" watchObservedRunningTime="2026-09-17 19:02:34.005266643 +0000 UTC m=+2005.993882892" Sep 17 19:02:34.953988 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:34.953944 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:02:34.954489 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:34.954214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:34.955431 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:34.955400 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:02:35.958145 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:35.958109 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:02:39.958664 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:39.958632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:02:39.959354 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:39.959329 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:02:40.962731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:40.962702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:02:40.963218 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:40.963195 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:02:49.959370 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:49.959334 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:02:50.964073 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:50.964020 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:02:59.959696 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:02:59.959657 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:03:00.963696 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:00.963658 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:03:09.959646 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:09.959606 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:03:10.964097 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:10.964051 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:03:19.959807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:19.959768 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:03:20.963346 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:20.963309 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:03:29.959804 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:29.959763 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:03:30.964030 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:30.963990 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:03:39.959609 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:39.959563 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused" Sep 17 19:03:40.963700 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:40.963657 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:03:45.996063 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:45.996033 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:03:50.963604 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:50.963569 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:03:57.340950 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.340912 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:03:57.341585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.341215 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" containerID="cri-o://9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744" gracePeriod=30 Sep 17 19:03:57.341585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.341250 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kube-rbac-proxy" containerID="cri-o://4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361" gracePeriod=30 Sep 17 19:03:57.394069 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.394041 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:03:57.394309 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.394268 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" containerID="cri-o://fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3" gracePeriod=30 Sep 17 19:03:57.394377 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:57.394285 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kube-rbac-proxy" containerID="cri-o://c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f" gracePeriod=30 Sep 17 19:03:58.315204 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:58.315169 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ef0330c-38d3-4146-9095-22786ba33155" containerID="c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f" exitCode=2 Sep 17 19:03:58.315389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:58.315240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerDied","Data":"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f"} Sep 17 19:03:58.317319 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:58.317294 2559 generic.go:356] "Generic (PLEG): container finished" podID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerID="4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361" exitCode=2 Sep 17 19:03:58.317437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:58.317324 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerDied","Data":"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361"} Sep 17 19:03:59.954191 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:03:59.954140 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused" Sep 17 19:04:00.959041 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:00.958995 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused" Sep 17 19:04:00.963328 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:00.963304 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 17 19:04:01.885435 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.885400 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:04:01.932386 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.932365 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:04:01.976442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976415 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls\") pod \"bd682c13-6191-42c7-bb8b-8cfd1724f652\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " Sep 17 19:04:01.976863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976470 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location\") pod \"bd682c13-6191-42c7-bb8b-8cfd1724f652\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " Sep 17 19:04:01.976863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976542 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config\") pod \"bd682c13-6191-42c7-bb8b-8cfd1724f652\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " Sep 17 19:04:01.976863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976564 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert\") pod \"bd682c13-6191-42c7-bb8b-8cfd1724f652\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " Sep 17 19:04:01.976863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976583 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w2ch\" (UniqueName: \"kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch\") pod \"bd682c13-6191-42c7-bb8b-8cfd1724f652\" (UID: \"bd682c13-6191-42c7-bb8b-8cfd1724f652\") " Sep 17 19:04:01.976863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd682c13-6191-42c7-bb8b-8cfd1724f652" (UID: "bd682c13-6191-42c7-bb8b-8cfd1724f652"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:04:01.977091 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976885 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd682c13-6191-42c7-bb8b-8cfd1724f652" (UID: "bd682c13-6191-42c7-bb8b-8cfd1724f652"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:04:01.977091 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.976892 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd682c13-6191-42c7-bb8b-8cfd1724f652" (UID: "bd682c13-6191-42c7-bb8b-8cfd1724f652"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:04:01.978651 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.978623 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch" (OuterVolumeSpecName: "kube-api-access-4w2ch") pod "bd682c13-6191-42c7-bb8b-8cfd1724f652" (UID: "bd682c13-6191-42c7-bb8b-8cfd1724f652"). InnerVolumeSpecName "kube-api-access-4w2ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:04:01.978729 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:01.978655 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd682c13-6191-42c7-bb8b-8cfd1724f652" (UID: "bd682c13-6191-42c7-bb8b-8cfd1724f652"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:04:02.077262 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077226 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpnvm\" (UniqueName: \"kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm\") pod \"1ef0330c-38d3-4146-9095-22786ba33155\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " Sep 17 19:04:02.077400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077277 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert\") pod \"1ef0330c-38d3-4146-9095-22786ba33155\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " Sep 17 19:04:02.077400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config\") pod \"1ef0330c-38d3-4146-9095-22786ba33155\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " Sep 17 19:04:02.077400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077345 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location\") pod \"1ef0330c-38d3-4146-9095-22786ba33155\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " Sep 17 19:04:02.077546 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077402 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls\") pod \"1ef0330c-38d3-4146-9095-22786ba33155\" (UID: \"1ef0330c-38d3-4146-9095-22786ba33155\") " Sep 17 19:04:02.077720 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077694 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd682c13-6191-42c7-bb8b-8cfd1724f652-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.077720 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077704 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ef0330c-38d3-4146-9095-22786ba33155" (UID: "1ef0330c-38d3-4146-9095-22786ba33155"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:04:02.077720 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077712 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ef0330c-38d3-4146-9095-22786ba33155" (UID: "1ef0330c-38d3-4146-9095-22786ba33155"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:04:02.077975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077723 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd682c13-6191-42c7-bb8b-8cfd1724f652-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.077975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077763 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.077975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077780 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd682c13-6191-42c7-bb8b-8cfd1724f652-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.077975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077797 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w2ch\" (UniqueName: \"kubernetes.io/projected/bd682c13-6191-42c7-bb8b-8cfd1724f652-kube-api-access-4w2ch\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.077975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.077924 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ef0330c-38d3-4146-9095-22786ba33155" (UID: "1ef0330c-38d3-4146-9095-22786ba33155"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:04:02.079404 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.079375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm" (OuterVolumeSpecName: "kube-api-access-zpnvm") pod "1ef0330c-38d3-4146-9095-22786ba33155" (UID: "1ef0330c-38d3-4146-9095-22786ba33155"). InnerVolumeSpecName "kube-api-access-zpnvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:04:02.079515 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.079430 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ef0330c-38d3-4146-9095-22786ba33155" (UID: "1ef0330c-38d3-4146-9095-22786ba33155"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:04:02.178355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.178288 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ef0330c-38d3-4146-9095-22786ba33155-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.178355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.178310 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnvm\" (UniqueName: \"kubernetes.io/projected/1ef0330c-38d3-4146-9095-22786ba33155-kube-api-access-zpnvm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.178355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.178322 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.178355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.178331 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ef0330c-38d3-4146-9095-22786ba33155-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.178355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.178339 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef0330c-38d3-4146-9095-22786ba33155-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:04:02.334963 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.334930 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ef0330c-38d3-4146-9095-22786ba33155" containerID="fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3" exitCode=0 Sep 17 19:04:02.335117 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.334990 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" Sep 17 19:04:02.335117 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.335024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerDied","Data":"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3"} Sep 17 19:04:02.335117 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.335098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55" event={"ID":"1ef0330c-38d3-4146-9095-22786ba33155","Type":"ContainerDied","Data":"ff18c14af55106196beb448c8219c0b4c06e5b59938f87af4882abecffedcd56"} Sep 17 19:04:02.335307 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.335136 2559 scope.go:117] "RemoveContainer" containerID="c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f" Sep 17 19:04:02.337181 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.337154 2559 generic.go:356] "Generic (PLEG): container finished" podID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerID="9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744" exitCode=0 Sep 17 19:04:02.337283 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.337193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerDied","Data":"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744"} Sep 17 19:04:02.337283 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.337222 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" Sep 17 19:04:02.337283 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.337242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn" event={"ID":"bd682c13-6191-42c7-bb8b-8cfd1724f652","Type":"ContainerDied","Data":"12cff204c59e97044fc334f05adeeefdb60003df2bab869f62a88bab06643b20"} Sep 17 19:04:02.343903 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.343886 2559 scope.go:117] "RemoveContainer" containerID="fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3" Sep 17 19:04:02.351088 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.351072 2559 scope.go:117] "RemoveContainer" containerID="f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15" Sep 17 19:04:02.360461 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.360425 2559 scope.go:117] "RemoveContainer" containerID="c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f" Sep 17 19:04:02.360929 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.360907 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f\": container with ID starting with c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f not found: ID does not exist" containerID="c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f" Sep 17 19:04:02.361009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.360935 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f"} err="failed to get container status \"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f\": rpc error: code = NotFound desc = could not find container \"c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f\": container with ID starting with c51f67908aefa1ce36f6bd8fd9f699fab9d9c390386d170973fe240a51c2830f not found: ID does not exist" Sep 17 19:04:02.361009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.360953 2559 scope.go:117] "RemoveContainer" containerID="fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3" Sep 17 19:04:02.361235 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.361216 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3\": container with ID starting with fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3 not found: ID does not exist" containerID="fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3" Sep 17 19:04:02.361325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.361239 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3"} err="failed to get container status \"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3\": rpc error: code = NotFound desc = could not find container \"fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3\": container with ID starting with fc3c95b6d2924d7c029924ff73fc00b9d7140e70c04a24b8afa9343f698096e3 not found: ID does not exist" Sep 17 19:04:02.361325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.361271 2559 scope.go:117] "RemoveContainer" containerID="f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15" Sep 17 19:04:02.361564 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.361548 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15\": container with ID starting with f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15 not found: ID does not exist" containerID="f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15" Sep 17 19:04:02.361622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.361568 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15"} err="failed to get container status \"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15\": rpc error: code = NotFound desc = could not find container \"f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15\": container with ID starting with f3ced9da42d2dfca80a3da1dfa2cc128b62a482cb63deb785f46c8794df7da15 not found: ID does not exist" Sep 17 19:04:02.361622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.361579 2559 scope.go:117] "RemoveContainer" containerID="4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361" Sep 17 19:04:02.365413 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.365389 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:04:02.368483 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.368462 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-mmqnn"] Sep 17 19:04:02.369282 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.369264 2559 scope.go:117] "RemoveContainer" containerID="9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744" Sep 17 19:04:02.376136 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.376120 2559 scope.go:117] "RemoveContainer" containerID="4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce" Sep 17 19:04:02.383344 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.383328 2559 scope.go:117] "RemoveContainer" containerID="4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361" Sep 17 19:04:02.383602 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.383582 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361\": container with ID starting with 4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361 not found: ID does not exist" containerID="4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361" Sep 17 19:04:02.383668 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.383609 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361"} err="failed to get container status \"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361\": rpc error: code = NotFound desc = could not find container \"4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361\": container with ID starting with 4579e64021ce26772c38f02773484dab84e12a2a2960b19967d54ef9f9f3a361 not found: ID does not exist" Sep 17 19:04:02.383668 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.383625 2559 scope.go:117] "RemoveContainer" containerID="9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744" Sep 17 19:04:02.383886 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.383868 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744\": container with ID starting with 9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744 not found: ID does not exist" containerID="9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744" Sep 17 19:04:02.383931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.383891 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744"} err="failed to get container status \"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744\": rpc error: code = NotFound desc = could not find container \"9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744\": container with ID starting with 9cdc4dc937b8b32fb38188cb03d4fe63c3b59f9313b85525beae68e124ff5744 not found: ID does not exist" Sep 17 19:04:02.383931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.383906 2559 scope.go:117] "RemoveContainer" containerID="4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce" Sep 17 19:04:02.384149 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:04:02.384124 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce\": container with ID starting with 4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce not found: ID does not exist" containerID="4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce" Sep 17 19:04:02.384207 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.384164 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce"} err="failed to get container status \"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce\": rpc error: code = NotFound desc = could not find container \"4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce\": container with ID starting with 4a19a0ad7cc36c5a6c2f632aac31846510ce163e719e7f103da90088490b77ce not found: ID does not exist" Sep 17 19:04:02.387367 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.387346 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:04:02.389639 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:02.389620 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-wlq55"] Sep 17 19:04:03.992589 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:03.992557 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef0330c-38d3-4146-9095-22786ba33155" path="/var/lib/kubelet/pods/1ef0330c-38d3-4146-9095-22786ba33155/volumes" Sep 17 19:04:03.993064 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:03.993051 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" path="/var/lib/kubelet/pods/bd682c13-6191-42c7-bb8b-8cfd1724f652/volumes" Sep 17 19:04:07.625419 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625387 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625715 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="storage-initializer" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625727 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="storage-initializer" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625739 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="storage-initializer" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625744 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="storage-initializer" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625753 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kube-rbac-proxy" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625760 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kube-rbac-proxy" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625767 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625772 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625778 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625790 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kube-rbac-proxy" Sep 17 19:04:07.625807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625795 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kube-rbac-proxy" Sep 17 19:04:07.626160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625840 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kserve-container" Sep 17 19:04:07.626160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625848 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kube-rbac-proxy" Sep 17 19:04:07.626160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625855 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd682c13-6191-42c7-bb8b-8cfd1724f652" containerName="kube-rbac-proxy" Sep 17 19:04:07.626160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.625862 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef0330c-38d3-4146-9095-22786ba33155" containerName="kserve-container" Sep 17 19:04:07.630059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.630044 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.632803 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.632779 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\"" Sep 17 19:04:07.632803 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.632788 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:04:07.633034 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.632809 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 19:04:07.633034 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.632808 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 19:04:07.633181 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.633170 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 19:04:07.633765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.633747 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\"" Sep 17 19:04:07.633856 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.633797 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 19:04:07.640820 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.640797 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:04:07.821642 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.821604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.821841 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.821652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.821841 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.821681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.821841 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.821769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.821841 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.821814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jbg\" (UniqueName: \"kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923132 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923267 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923267 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923267 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923267 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jbg\" (UniqueName: \"kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923659 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.923932 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.923915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.925683 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.925663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.932846 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.932824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jbg\" (UniqueName: \"kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg\") pod \"isvc-lightgbm-predictor-54f85ddf88-6js4s\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:07.939649 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:07.939628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:08.211478 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:08.211437 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:04:08.213124 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:04:08.213096 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1b426a_26a1_46fb_8133_cc82182e6005.slice/crio-edf4f8fd51eb6e488b8e6dde55148a3634665317484c7d49df03c333833e44b6 WatchSource:0}: Error finding container edf4f8fd51eb6e488b8e6dde55148a3634665317484c7d49df03c333833e44b6: Status 404 returned error can't find the container with id edf4f8fd51eb6e488b8e6dde55148a3634665317484c7d49df03c333833e44b6 Sep 17 19:04:08.365134 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:08.365096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerStarted","Data":"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99"} Sep 17 19:04:08.365318 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:08.365162 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerStarted","Data":"edf4f8fd51eb6e488b8e6dde55148a3634665317484c7d49df03c333833e44b6"} Sep 17 19:04:10.063942 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:10.063911 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:04:10.065572 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:10.065550 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:04:10.072227 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:10.072203 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:04:10.073626 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:10.073604 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:04:12.383680 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:12.383592 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerID="ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99" exitCode=0 Sep 17 19:04:12.383680 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:12.383645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerDied","Data":"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99"} Sep 17 19:04:19.420260 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:19.420233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerStarted","Data":"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb"} Sep 17 19:04:20.426127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:20.426086 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerStarted","Data":"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a"} Sep 17 19:04:20.426594 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:20.426155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:20.426594 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:20.426185 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:20.427784 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:20.427756 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:04:20.445923 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:20.445879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podStartSLOduration=2.675867761 podStartE2EDuration="13.445866527s" totalImagesPullingTime="6.928590801s" totalInitContainerRuntime="3.841407965s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:20.445342555 +0000 UTC m=+2112.433958804" watchObservedRunningTime="2026-09-17 19:04:20.445866527 +0000 UTC m=+2112.434482775" Sep 17 19:04:21.430676 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:21.430641 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:04:26.442022 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:26.441990 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:04:26.442615 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:26.442589 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:04:36.443276 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:36.443236 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:04:46.442830 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:46.442789 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:04:56.443157 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:04:56.443076 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:06.443353 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:06.443311 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:16.443539 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:16.443493 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:26.443121 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:26.443073 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:36.428010 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.427971 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:05:36.430428 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.430412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.433237 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.433212 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\"" Sep 17 19:05:36.433390 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.433240 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\"" Sep 17 19:05:36.443147 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.443125 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:36.447322 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.447301 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:05:36.519003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.518978 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.519003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.519005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.519175 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.519025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.519175 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.519094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.519175 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.519120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc4l\" (UniqueName: \"kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.619655 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.619627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc4l\" (UniqueName: \"kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.619820 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.619671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.619820 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.619688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.619820 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.619802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.620002 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.619901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.620275 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.620251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.620408 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.620392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.620486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.620467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.622279 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.622250 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.634312 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.634278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc4l\" (UniqueName: \"kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:36.742974 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:36.742895 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:37.020373 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:37.020345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:05:37.022671 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:05:37.022634 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8315ca2e_edef_416b_93d8_d3402aaad14a.slice/crio-4572670ac7bf3b96f2729de0153d5a9f252a67b6f5b4ea4b3c29b82e26580969 WatchSource:0}: Error finding container 4572670ac7bf3b96f2729de0153d5a9f252a67b6f5b4ea4b3c29b82e26580969: Status 404 returned error can't find the container with id 4572670ac7bf3b96f2729de0153d5a9f252a67b6f5b4ea4b3c29b82e26580969 Sep 17 19:05:37.745722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:37.745680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerStarted","Data":"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310"} Sep 17 19:05:37.746088 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:37.745742 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerStarted","Data":"4572670ac7bf3b96f2729de0153d5a9f252a67b6f5b4ea4b3c29b82e26580969"} Sep 17 19:05:41.765633 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:41.765590 2559 generic.go:356] "Generic (PLEG): container finished" podID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerID="ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310" exitCode=0 Sep 17 19:05:41.766046 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:41.765648 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerDied","Data":"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310"} Sep 17 19:05:42.772206 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:42.772164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerStarted","Data":"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211"} Sep 17 19:05:42.772619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:42.772235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerStarted","Data":"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05"} Sep 17 19:05:42.772619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:42.772442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:42.797746 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:42.797695 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" podStartSLOduration=3.090452814 podStartE2EDuration="6.797681305s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707228491s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:42.796881381 +0000 UTC m=+2194.785497629" watchObservedRunningTime="2026-09-17 19:05:42.797681305 +0000 UTC m=+2194.786297553" Sep 17 19:05:43.776355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:43.776318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:46.442750 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:46.442711 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused" Sep 17 19:05:49.786927 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:49.786900 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:05:55.992644 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:55.992613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:05:58.191272 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:58.191238 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:05:58.191719 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:58.191536 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" containerID="cri-o://a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb" gracePeriod=30 Sep 17 19:05:58.191719 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:58.191607 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kube-rbac-proxy" containerID="cri-o://36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a" gracePeriod=30 Sep 17 19:05:58.842760 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:58.842726 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerID="36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a" exitCode=2 Sep 17 19:05:58.842925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:05:58.842796 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerDied","Data":"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a"} Sep 17 19:06:01.431153 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:01.431115 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused" Sep 17 19:06:03.398638 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.398601 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:06:03.401220 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.401200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.404193 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.404172 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\"" Sep 17 19:06:03.404310 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.404221 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:06:03.416126 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.416093 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:06:03.543848 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.543815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k89\" (UniqueName: \"kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.544009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.543854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.544009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.543902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.544009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.543933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.544009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.543952 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.644800 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.644756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.644800 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.644804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645040 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.644855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k89\" (UniqueName: \"kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645040 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.644872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645040 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.644905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645040 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:03.644986 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found Sep 17 19:06:03.645203 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:03.645090 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls podName:d24ef56d-a6c4-411c-95af-678a42de1cdd nodeName:}" failed. No retries permitted until 2026-09-17 19:06:04.145068144 +0000 UTC m=+2216.133684375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found Sep 17 19:06:03.645362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.645340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.645568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.645630 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.645607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.655370 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.655349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k89\" (UniqueName: \"kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:03.847682 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.847652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:06:03.866485 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.866430 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerID="a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb" exitCode=0 Sep 17 19:06:03.866639 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.866483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerDied","Data":"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb"} Sep 17 19:06:03.866639 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.866505 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" Sep 17 19:06:03.866639 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.866568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s" event={"ID":"5d1b426a-26a1-46fb-8133-cc82182e6005","Type":"ContainerDied","Data":"edf4f8fd51eb6e488b8e6dde55148a3634665317484c7d49df03c333833e44b6"} Sep 17 19:06:03.866639 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.866603 2559 scope.go:117] "RemoveContainer" containerID="36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a" Sep 17 19:06:03.874514 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.874493 2559 scope.go:117] "RemoveContainer" containerID="a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb" Sep 17 19:06:03.881752 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.881736 2559 scope.go:117] "RemoveContainer" containerID="ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99" Sep 17 19:06:03.888794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.888766 2559 scope.go:117] "RemoveContainer" containerID="36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a" Sep 17 19:06:03.889057 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:03.889038 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a\": container with ID starting with 36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a not found: ID does not exist" containerID="36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a" Sep 17 19:06:03.889119 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.889066 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a"} err="failed to get container status \"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a\": rpc error: code = NotFound desc = could not find container \"36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a\": container with ID starting with 36d507d08aeead320c39f47b2a56fe548c1b9a02ebaefdba4c43215d9129f90a not found: ID does not exist" Sep 17 19:06:03.889119 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.889082 2559 scope.go:117] "RemoveContainer" containerID="a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb" Sep 17 19:06:03.889341 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:03.889323 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb\": container with ID starting with a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb not found: ID does not exist" containerID="a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb" Sep 17 19:06:03.889382 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.889347 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb"} err="failed to get container status \"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb\": rpc error: code = NotFound desc = could not find container \"a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb\": container with ID starting with a9885c638d32471358d0159693210947c2301eda06af6197e1b2c174340b53cb not found: ID does not exist" Sep 17 19:06:03.889382 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.889363 2559 scope.go:117] "RemoveContainer" containerID="ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99" Sep 17 19:06:03.889614 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:03.889596 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99\": container with ID starting with ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99 not found: ID does not exist" containerID="ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99" Sep 17 19:06:03.889663 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.889620 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99"} err="failed to get container status \"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99\": rpc error: code = NotFound desc = could not find container \"ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99\": container with ID starting with ee49c2707cb964b8d826dd8b3a3552eaeb83b8e765840204b1c1ba09ab513c99 not found: ID does not exist" Sep 17 19:06:03.946829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.946755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location\") pod \"5d1b426a-26a1-46fb-8133-cc82182e6005\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " Sep 17 19:06:03.946829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.946794 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls\") pod \"5d1b426a-26a1-46fb-8133-cc82182e6005\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " Sep 17 19:06:03.947024 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.946835 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config\") pod \"5d1b426a-26a1-46fb-8133-cc82182e6005\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " Sep 17 19:06:03.947024 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.946928 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert\") pod \"5d1b426a-26a1-46fb-8133-cc82182e6005\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " Sep 17 19:06:03.947024 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.946964 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7jbg\" (UniqueName: \"kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg\") pod \"5d1b426a-26a1-46fb-8133-cc82182e6005\" (UID: \"5d1b426a-26a1-46fb-8133-cc82182e6005\") " Sep 17 19:06:03.947152 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.947139 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d1b426a-26a1-46fb-8133-cc82182e6005" (UID: "5d1b426a-26a1-46fb-8133-cc82182e6005"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:06:03.947221 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.947206 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d1b426a-26a1-46fb-8133-cc82182e6005-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:03.947307 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.947286 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d1b426a-26a1-46fb-8133-cc82182e6005" (UID: "5d1b426a-26a1-46fb-8133-cc82182e6005"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:06:03.947350 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.947284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d1b426a-26a1-46fb-8133-cc82182e6005" (UID: "5d1b426a-26a1-46fb-8133-cc82182e6005"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:06:03.949033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.949010 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d1b426a-26a1-46fb-8133-cc82182e6005" (UID: "5d1b426a-26a1-46fb-8133-cc82182e6005"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:06:03.949563 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:03.949540 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg" (OuterVolumeSpecName: "kube-api-access-v7jbg") pod "5d1b426a-26a1-46fb-8133-cc82182e6005" (UID: "5d1b426a-26a1-46fb-8133-cc82182e6005"). InnerVolumeSpecName "kube-api-access-v7jbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:06:04.048181 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.048157 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d1b426a-26a1-46fb-8133-cc82182e6005-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:04.048181 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.048179 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:04.048338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.048189 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d1b426a-26a1-46fb-8133-cc82182e6005-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:04.048338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.048198 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7jbg\" (UniqueName: \"kubernetes.io/projected/5d1b426a-26a1-46fb-8133-cc82182e6005-kube-api-access-v7jbg\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:04.148821 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.148772 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:04.151602 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.151579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:04.187259 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.187232 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:06:04.196278 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.196251 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6js4s"] Sep 17 19:06:04.311665 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.311577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:04.593878 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.593848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:06:04.596344 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:06:04.596314 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24ef56d_a6c4_411c_95af_678a42de1cdd.slice/crio-3dbd6b4bf283c2930a141fea7195e6b0bbfd28a1b17f3dd75ad66a64b237c3f3 WatchSource:0}: Error finding container 3dbd6b4bf283c2930a141fea7195e6b0bbfd28a1b17f3dd75ad66a64b237c3f3: Status 404 returned error can't find the container with id 3dbd6b4bf283c2930a141fea7195e6b0bbfd28a1b17f3dd75ad66a64b237c3f3 Sep 17 19:06:04.873005 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.872895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerStarted","Data":"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd"} Sep 17 19:06:04.873005 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:04.872974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerStarted","Data":"3dbd6b4bf283c2930a141fea7195e6b0bbfd28a1b17f3dd75ad66a64b237c3f3"} Sep 17 19:06:05.992430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:05.992399 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" path="/var/lib/kubelet/pods/5d1b426a-26a1-46fb-8133-cc82182e6005/volumes" Sep 17 19:06:08.893786 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:08.893751 2559 generic.go:356] "Generic (PLEG): container finished" podID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerID="47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd" exitCode=0 Sep 17 19:06:08.894153 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:08.893812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerDied","Data":"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd"} Sep 17 19:06:09.900386 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.900351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerStarted","Data":"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6"} Sep 17 19:06:09.900798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.900412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerStarted","Data":"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249"} Sep 17 19:06:09.900798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.900710 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:09.900892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.900872 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:09.902192 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.902166 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:09.925001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:09.924957 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podStartSLOduration=3.041564195 podStartE2EDuration="6.92494443s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.883380235s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:09.922403176 +0000 UTC m=+2221.911019425" watchObservedRunningTime="2026-09-17 19:06:09.92494443 +0000 UTC m=+2221.913560679" Sep 17 19:06:10.905648 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:10.905592 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:15.911833 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:15.911802 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:06:15.912345 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:15.912320 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:19.791517 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:19.791482 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:06:25.913323 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:25.913234 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:26.557687 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:26.557649 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:06:26.557970 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:26.557939 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kserve-container" containerID="cri-o://fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05" gracePeriod=30 Sep 17 19:06:26.558057 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:26.557942 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kube-rbac-proxy" containerID="cri-o://ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211" gracePeriod=30 Sep 17 19:06:26.975049 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:26.975016 2559 generic.go:356] "Generic (PLEG): container finished" podID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerID="ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211" exitCode=2 Sep 17 19:06:26.975429 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:26.975089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerDied","Data":"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211"} Sep 17 19:06:29.782028 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:29.781981 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused" Sep 17 19:06:29.787563 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:29.787531 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.51:8080: connect: connection refused" Sep 17 19:06:31.763464 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763771 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kube-rbac-proxy" Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kube-rbac-proxy" Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763801 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="storage-initializer" Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763806 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="storage-initializer" Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763815 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" Sep 17 19:06:31.763829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763821 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" Sep 17 19:06:31.764023 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763892 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kube-rbac-proxy" Sep 17 19:06:31.764023 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.763899 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1b426a-26a1-46fb-8133-cc82182e6005" containerName="kserve-container" Sep 17 19:06:31.766042 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.766025 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.769267 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.769245 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\"" Sep 17 19:06:31.769408 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.769248 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\"" Sep 17 19:06:31.779782 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.779755 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:06:31.890366 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.890329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.890587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.890380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.890587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.890399 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6znf\" (UniqueName: \"kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.890587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.890507 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.890587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.890544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991271 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6znf\" (UniqueName: \"kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991437 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:31.991411 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found Sep 17 19:06:31.991673 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.991673 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:31.991508 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls podName:32aa9b04-9c9a-4acc-a568-6ea7d869bd9a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:32.491487066 +0000 UTC m=+2244.480103296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found Sep 17 19:06:31.991770 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.991751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.992064 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.992041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:31.992118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:31.992100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:32.001269 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:32.001247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6znf\" (UniqueName: \"kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:32.498006 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:32.497972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:32.500510 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:32.500488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fsx76\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:32.675999 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:32.675947 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:32.946611 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:32.946586 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:06:32.949198 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:06:32.949166 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aa9b04_9c9a_4acc_a568_6ea7d869bd9a.slice/crio-c0148e12fb3b2b8f76feb01c0cfbae366fa372c1851bdec45b1e8f084e27bff1 WatchSource:0}: Error finding container c0148e12fb3b2b8f76feb01c0cfbae366fa372c1851bdec45b1e8f084e27bff1: Status 404 returned error can't find the container with id c0148e12fb3b2b8f76feb01c0cfbae366fa372c1851bdec45b1e8f084e27bff1 Sep 17 19:06:33.000731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.000701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerStarted","Data":"c0148e12fb3b2b8f76feb01c0cfbae366fa372c1851bdec45b1e8f084e27bff1"} Sep 17 19:06:33.314946 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.314918 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:06:33.408049 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408016 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls\") pod \"8315ca2e-edef-416b-93d8-d3402aaad14a\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " Sep 17 19:06:33.408237 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408114 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert\") pod \"8315ca2e-edef-416b-93d8-d3402aaad14a\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " Sep 17 19:06:33.408237 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408214 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config\") pod \"8315ca2e-edef-416b-93d8-d3402aaad14a\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " Sep 17 19:06:33.408340 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408268 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbc4l\" (UniqueName: \"kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l\") pod \"8315ca2e-edef-416b-93d8-d3402aaad14a\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " Sep 17 19:06:33.408340 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408296 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location\") pod \"8315ca2e-edef-416b-93d8-d3402aaad14a\" (UID: \"8315ca2e-edef-416b-93d8-d3402aaad14a\") " Sep 17 19:06:33.408500 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8315ca2e-edef-416b-93d8-d3402aaad14a" (UID: "8315ca2e-edef-416b-93d8-d3402aaad14a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:06:33.408598 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408583 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:33.408660 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408544 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8315ca2e-edef-416b-93d8-d3402aaad14a" (UID: "8315ca2e-edef-416b-93d8-d3402aaad14a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:06:33.408704 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.408664 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8315ca2e-edef-416b-93d8-d3402aaad14a" (UID: "8315ca2e-edef-416b-93d8-d3402aaad14a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:06:33.410257 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.410223 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8315ca2e-edef-416b-93d8-d3402aaad14a" (UID: "8315ca2e-edef-416b-93d8-d3402aaad14a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:06:33.410667 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.410645 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l" (OuterVolumeSpecName: "kube-api-access-kbc4l") pod "8315ca2e-edef-416b-93d8-d3402aaad14a" (UID: "8315ca2e-edef-416b-93d8-d3402aaad14a"). InnerVolumeSpecName "kube-api-access-kbc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:06:33.509865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.509792 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8315ca2e-edef-416b-93d8-d3402aaad14a-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:33.509865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.509819 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:33.509865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.509831 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc4l\" (UniqueName: \"kubernetes.io/projected/8315ca2e-edef-416b-93d8-d3402aaad14a-kube-api-access-kbc4l\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:33.509865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:33.509840 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8315ca2e-edef-416b-93d8-d3402aaad14a-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:06:34.007394 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.007358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerStarted","Data":"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8"} Sep 17 19:06:34.009279 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.009255 2559 generic.go:356] "Generic (PLEG): container finished" podID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerID="fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05" exitCode=0 Sep 17 19:06:34.009400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.009318 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" Sep 17 19:06:34.009400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.009332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerDied","Data":"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05"} Sep 17 19:06:34.009400 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.009391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r" event={"ID":"8315ca2e-edef-416b-93d8-d3402aaad14a","Type":"ContainerDied","Data":"4572670ac7bf3b96f2729de0153d5a9f252a67b6f5b4ea4b3c29b82e26580969"} Sep 17 19:06:34.009539 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.009431 2559 scope.go:117] "RemoveContainer" containerID="ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211" Sep 17 19:06:34.017766 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.017752 2559 scope.go:117] "RemoveContainer" containerID="fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05" Sep 17 19:06:34.024910 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.024895 2559 scope.go:117] "RemoveContainer" containerID="ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310" Sep 17 19:06:34.033063 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033043 2559 scope.go:117] "RemoveContainer" containerID="ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211" Sep 17 19:06:34.033317 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:34.033298 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211\": container with ID starting with ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211 not found: ID does not exist" containerID="ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211" Sep 17 19:06:34.033415 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033321 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211"} err="failed to get container status \"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211\": rpc error: code = NotFound desc = could not find container \"ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211\": container with ID starting with ffd3a5f330ececc3e3593b725b6f13bd2a4d6ee32722604d95e1804b8e77a211 not found: ID does not exist" Sep 17 19:06:34.033415 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033337 2559 scope.go:117] "RemoveContainer" containerID="fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05" Sep 17 19:06:34.033579 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:34.033562 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05\": container with ID starting with fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05 not found: ID does not exist" containerID="fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05" Sep 17 19:06:34.033622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033578 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05"} err="failed to get container status \"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05\": rpc error: code = NotFound desc = could not find container \"fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05\": container with ID starting with fa82b3f81942b0330550275202d82a96a9474b8965dfe2ac716f136ab1b7ed05 not found: ID does not exist" Sep 17 19:06:34.033622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033590 2559 scope.go:117] "RemoveContainer" containerID="ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310" Sep 17 19:06:34.033839 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:06:34.033821 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310\": container with ID starting with ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310 not found: ID does not exist" containerID="ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310" Sep 17 19:06:34.033878 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.033844 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310"} err="failed to get container status \"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310\": rpc error: code = NotFound desc = could not find container \"ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310\": container with ID starting with ff3fc13b08aee72983559309d7e5f320ea2056767d91b2cb227e4088340cf310 not found: ID does not exist" Sep 17 19:06:34.045047 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.045016 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:06:34.050990 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:34.050966 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-pbp5r"] Sep 17 19:06:35.912414 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:35.912381 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:35.992133 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:35.992101 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" path="/var/lib/kubelet/pods/8315ca2e-edef-416b-93d8-d3402aaad14a/volumes" Sep 17 19:06:37.027734 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:37.027660 2559 generic.go:356] "Generic (PLEG): container finished" podID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerID="9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8" exitCode=0 Sep 17 19:06:37.028076 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:37.027732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerDied","Data":"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8"} Sep 17 19:06:38.034009 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:38.033977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerStarted","Data":"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179"} Sep 17 19:06:38.034364 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:38.034029 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerStarted","Data":"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae"} Sep 17 19:06:38.034364 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:38.034205 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:39.039099 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:39.039063 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:45.049573 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:45.049539 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:06:45.072633 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:45.072492 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" podStartSLOduration=10.348657754 podStartE2EDuration="14.072473188s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.723815434s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:38.085526579 +0000 UTC m=+2250.074142827" watchObservedRunningTime="2026-09-17 19:06:45.072473188 +0000 UTC m=+2257.061089432" Sep 17 19:06:45.912634 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:45.912596 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:06:55.912693 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:06:55.912653 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:05.912758 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:05.912716 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:15.053907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:15.053876 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:07:15.912442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:15.912405 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:21.736990 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:21.736959 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:07:21.737378 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:21.737216 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kserve-container" containerID="cri-o://58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae" gracePeriod=30 Sep 17 19:07:21.737378 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:21.737273 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kube-rbac-proxy" containerID="cri-o://ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179" gracePeriod=30 Sep 17 19:07:22.237739 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:22.237708 2559 generic.go:356] "Generic (PLEG): container finished" podID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerID="ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179" exitCode=2 Sep 17 19:07:22.237898 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:22.237754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerDied","Data":"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179"} Sep 17 19:07:25.044304 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:25.044265 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused" Sep 17 19:07:25.050230 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:25.050183 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.53:8080: connect: connection refused" Sep 17 19:07:25.912638 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:25.912598 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:26.927849 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.927810 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928126 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="storage-initializer" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928146 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="storage-initializer" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928158 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kserve-container" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928164 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kserve-container" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928177 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kube-rbac-proxy" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928182 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kube-rbac-proxy" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928246 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kube-rbac-proxy" Sep 17 19:07:26.928300 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.928255 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8315ca2e-edef-416b-93d8-d3402aaad14a" containerName="kserve-container" Sep 17 19:07:26.930166 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.930149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:26.933040 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.933016 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\"" Sep 17 19:07:26.933177 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.933060 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:07:26.943966 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:26.943945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:07:27.042568 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.042528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.042568 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.042573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.042822 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.042634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndtj\" (UniqueName: \"kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.042822 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.042690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.042931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.042815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143221 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndtj\" (UniqueName: \"kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143795 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.143969 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.144047 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.143975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.145634 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.145618 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.152872 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.152847 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndtj\" (UniqueName: \"kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-4hwx6\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.239733 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.239659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:27.512879 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.512827 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:07:27.515148 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:07:27.515115 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf911a896_c961_4700_b1bc_6b3a31d776ee.slice/crio-d2c0a6822af50da4416b5e1b0679cf29e4bcc6566f57215af37947b5d027a4e8 WatchSource:0}: Error finding container d2c0a6822af50da4416b5e1b0679cf29e4bcc6566f57215af37947b5d027a4e8: Status 404 returned error can't find the container with id d2c0a6822af50da4416b5e1b0679cf29e4bcc6566f57215af37947b5d027a4e8 Sep 17 19:07:27.981896 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:27.981870 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:07:28.051130 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051097 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") pod \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " Sep 17 19:07:28.051130 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config\") pod \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " Sep 17 19:07:28.051351 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051151 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert\") pod \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " Sep 17 19:07:28.051351 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051232 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6znf\" (UniqueName: \"kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf\") pod \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " Sep 17 19:07:28.051351 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051289 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location\") pod \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\" (UID: \"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a\") " Sep 17 19:07:28.051779 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051589 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:07:28.051779 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:07:28.051967 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.051852 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:07:28.053813 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.053794 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:07:28.053998 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.053978 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf" (OuterVolumeSpecName: "kube-api-access-t6znf") pod "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" (UID: "32aa9b04-9c9a-4acc-a568-6ea7d869bd9a"). InnerVolumeSpecName "kube-api-access-t6znf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:07:28.152315 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.152278 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6znf\" (UniqueName: \"kubernetes.io/projected/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-api-access-t6znf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:28.152315 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.152307 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:28.152523 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.152321 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:28.152523 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.152336 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:28.152523 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.152347 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:28.267423 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.267384 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerStarted","Data":"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2"} Sep 17 19:07:28.267634 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.267498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerStarted","Data":"d2c0a6822af50da4416b5e1b0679cf29e4bcc6566f57215af37947b5d027a4e8"} Sep 17 19:07:28.269349 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.269324 2559 generic.go:356] "Generic (PLEG): container finished" podID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerID="58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae" exitCode=0 Sep 17 19:07:28.269487 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.269382 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" Sep 17 19:07:28.269487 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.269403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerDied","Data":"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae"} Sep 17 19:07:28.269487 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.269464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76" event={"ID":"32aa9b04-9c9a-4acc-a568-6ea7d869bd9a","Type":"ContainerDied","Data":"c0148e12fb3b2b8f76feb01c0cfbae366fa372c1851bdec45b1e8f084e27bff1"} Sep 17 19:07:28.269595 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.269491 2559 scope.go:117] "RemoveContainer" containerID="ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179" Sep 17 19:07:28.278089 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.278073 2559 scope.go:117] "RemoveContainer" containerID="58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae" Sep 17 19:07:28.285655 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.285634 2559 scope.go:117] "RemoveContainer" containerID="9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8" Sep 17 19:07:28.293321 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.293301 2559 scope.go:117] "RemoveContainer" containerID="ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179" Sep 17 19:07:28.293603 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:28.293581 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179\": container with ID starting with ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179 not found: ID does not exist" containerID="ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179" Sep 17 19:07:28.293672 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.293608 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179"} err="failed to get container status \"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179\": rpc error: code = NotFound desc = could not find container \"ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179\": container with ID starting with ae731b142b06702499c89b6da6791bb5c90297fbb56efaaf47061c9ee85f9179 not found: ID does not exist" Sep 17 19:07:28.293672 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.293626 2559 scope.go:117] "RemoveContainer" containerID="58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae" Sep 17 19:07:28.293866 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:28.293847 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae\": container with ID starting with 58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae not found: ID does not exist" containerID="58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae" Sep 17 19:07:28.293907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.293873 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae"} err="failed to get container status \"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae\": rpc error: code = NotFound desc = could not find container \"58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae\": container with ID starting with 58c50c8288ab6f9b51e4b19868540585f6a4651605cf51a14225cc2698e038ae not found: ID does not exist" Sep 17 19:07:28.293907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.293889 2559 scope.go:117] "RemoveContainer" containerID="9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8" Sep 17 19:07:28.294126 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:28.294106 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8\": container with ID starting with 9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8 not found: ID does not exist" containerID="9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8" Sep 17 19:07:28.294188 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.294129 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8"} err="failed to get container status \"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8\": rpc error: code = NotFound desc = could not find container \"9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8\": container with ID starting with 9c034cd082f4cce84a658e8af70e0ff6837fac0f8268f369c6e465ec65bd2ce8 not found: ID does not exist" Sep 17 19:07:28.305874 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.305855 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:07:28.311304 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:28.311284 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fsx76"] Sep 17 19:07:29.992946 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:29.992916 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" path="/var/lib/kubelet/pods/32aa9b04-9c9a-4acc-a568-6ea7d869bd9a/volumes" Sep 17 19:07:30.988217 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:30.988176 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:32.289880 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:32.289848 2559 generic.go:356] "Generic (PLEG): container finished" podID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerID="4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2" exitCode=0 Sep 17 19:07:32.290366 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:32.289924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerDied","Data":"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2"} Sep 17 19:07:32.291211 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:32.291196 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 19:07:40.989530 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:40.989495 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:07:45.460357 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:45.460326 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:07:45.460830 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:45.460638 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" containerID="cri-o://96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" gracePeriod=30 Sep 17 19:07:45.460830 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:45.460666 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" containerID="cri-o://bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" gracePeriod=30 Sep 17 19:07:45.906463 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:45.906420 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused" Sep 17 19:07:50.672930 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.672896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673278 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="storage-initializer" Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673291 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="storage-initializer" Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673299 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kserve-container" Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673305 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kserve-container" Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673320 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kube-rbac-proxy" Sep 17 19:07:50.673336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673326 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kube-rbac-proxy" Sep 17 19:07:50.673597 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673388 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kserve-container" Sep 17 19:07:50.673597 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.673401 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aa9b04-9c9a-4acc-a568-6ea7d869bd9a" containerName="kube-rbac-proxy" Sep 17 19:07:50.722513 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.722471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:07:50.722671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.722577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.725336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.725299 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:07:50.725502 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.725439 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\"" Sep 17 19:07:50.754440 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.754415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.754601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.754475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.754601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.754521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.754601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.754589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsskq\" (UniqueName: \"kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.754767 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.754612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855278 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsskq\" (UniqueName: \"kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855471 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855471 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855471 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855471 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.855690 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:50.855581 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found Sep 17 19:07:50.855690 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:50.855644 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls podName:c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd nodeName:}" failed. No retries permitted until 2026-09-17 19:07:51.355623008 +0000 UTC m=+2323.344239239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found Sep 17 19:07:50.855816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.855772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.856253 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.856228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.856350 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.856230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.865052 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.865029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsskq\" (UniqueName: \"kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:50.906678 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.906643 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused" Sep 17 19:07:50.988819 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:50.988744 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused" Sep 17 19:07:51.360728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:51.360644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:51.363279 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:51.363252 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:51.634794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:51.634702 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:52.762050 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.761989 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:07:52.770801 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.770774 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location\") pod \"d24ef56d-a6c4-411c-95af-678a42de1cdd\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " Sep 17 19:07:52.770918 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.770857 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") pod \"d24ef56d-a6c4-411c-95af-678a42de1cdd\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " Sep 17 19:07:52.770918 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.770884 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4k89\" (UniqueName: \"kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89\") pod \"d24ef56d-a6c4-411c-95af-678a42de1cdd\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " Sep 17 19:07:52.770918 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.770909 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config\") pod \"d24ef56d-a6c4-411c-95af-678a42de1cdd\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " Sep 17 19:07:52.771127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.771098 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d24ef56d-a6c4-411c-95af-678a42de1cdd" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:07:52.771350 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.771326 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d24ef56d-a6c4-411c-95af-678a42de1cdd" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:07:52.772931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.772908 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89" (OuterVolumeSpecName: "kube-api-access-n4k89") pod "d24ef56d-a6c4-411c-95af-678a42de1cdd" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd"). InnerVolumeSpecName "kube-api-access-n4k89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:07:52.773227 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.773210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d24ef56d-a6c4-411c-95af-678a42de1cdd" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:07:52.871360 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871341 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert\") pod \"d24ef56d-a6c4-411c-95af-678a42de1cdd\" (UID: \"d24ef56d-a6c4-411c-95af-678a42de1cdd\") " Sep 17 19:07:52.871556 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871542 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24ef56d-a6c4-411c-95af-678a42de1cdd-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:52.871623 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871561 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24ef56d-a6c4-411c-95af-678a42de1cdd-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:52.871623 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871574 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k89\" (UniqueName: \"kubernetes.io/projected/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-api-access-n4k89\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:52.871623 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871589 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:52.871727 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.871641 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d24ef56d-a6c4-411c-95af-678a42de1cdd" (UID: "d24ef56d-a6c4-411c-95af-678a42de1cdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:07:52.962168 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.962146 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:07:52.964496 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:07:52.964469 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40b7e1d_da67_4cc9_b0c0_f1fa6684e7fd.slice/crio-09a85041b14ce9d09bdece91d5ae2cb1ac0e5836f7c75d9f4569a7fbab2f4e27 WatchSource:0}: Error finding container 09a85041b14ce9d09bdece91d5ae2cb1ac0e5836f7c75d9f4569a7fbab2f4e27: Status 404 returned error can't find the container with id 09a85041b14ce9d09bdece91d5ae2cb1ac0e5836f7c75d9f4569a7fbab2f4e27 Sep 17 19:07:52.972679 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:52.972656 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24ef56d-a6c4-411c-95af-678a42de1cdd-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:07:53.401402 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.401368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerStarted","Data":"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2"} Sep 17 19:07:53.401606 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.401443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerStarted","Data":"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4"} Sep 17 19:07:53.401742 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.401722 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:53.403056 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.403028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerStarted","Data":"0031a16587f55ea9184b9887b970563f6704cd14f6abfc770bf17ccb784bcd19"} Sep 17 19:07:53.403182 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.403073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerStarted","Data":"09a85041b14ce9d09bdece91d5ae2cb1ac0e5836f7c75d9f4569a7fbab2f4e27"} Sep 17 19:07:53.405146 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405122 2559 generic.go:356] "Generic (PLEG): container finished" podID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerID="bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" exitCode=2 Sep 17 19:07:53.405146 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405142 2559 generic.go:356] "Generic (PLEG): container finished" podID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerID="96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" exitCode=0 Sep 17 19:07:53.405338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerDied","Data":"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6"} Sep 17 19:07:53.405338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405174 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" Sep 17 19:07:53.405338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerDied","Data":"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249"} Sep 17 19:07:53.405338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc" event={"ID":"d24ef56d-a6c4-411c-95af-678a42de1cdd","Type":"ContainerDied","Data":"3dbd6b4bf283c2930a141fea7195e6b0bbfd28a1b17f3dd75ad66a64b237c3f3"} Sep 17 19:07:53.405338 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.405290 2559 scope.go:117] "RemoveContainer" containerID="bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" Sep 17 19:07:53.414194 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.414175 2559 scope.go:117] "RemoveContainer" containerID="96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" Sep 17 19:07:53.420569 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.420527 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podStartSLOduration=2.91371141 podStartE2EDuration="27.420512416s" totalImagesPullingTime="20.530027851s" totalInitContainerRuntime="3.976773155s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:53.419118147 +0000 UTC m=+2325.407734396" watchObservedRunningTime="2026-09-17 19:07:53.420512416 +0000 UTC m=+2325.409128665" Sep 17 19:07:53.422625 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.422605 2559 scope.go:117] "RemoveContainer" containerID="47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd" Sep 17 19:07:53.429816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.429787 2559 scope.go:117] "RemoveContainer" containerID="bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" Sep 17 19:07:53.430124 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:53.430096 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6\": container with ID starting with bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6 not found: ID does not exist" containerID="bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" Sep 17 19:07:53.430198 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430136 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6"} err="failed to get container status \"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6\": rpc error: code = NotFound desc = could not find container \"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6\": container with ID starting with bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6 not found: ID does not exist" Sep 17 19:07:53.430198 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430160 2559 scope.go:117] "RemoveContainer" containerID="96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" Sep 17 19:07:53.430461 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:53.430432 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249\": container with ID starting with 96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249 not found: ID does not exist" containerID="96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" Sep 17 19:07:53.430506 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430471 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249"} err="failed to get container status \"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249\": rpc error: code = NotFound desc = could not find container \"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249\": container with ID starting with 96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249 not found: ID does not exist" Sep 17 19:07:53.430506 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430492 2559 scope.go:117] "RemoveContainer" containerID="47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd" Sep 17 19:07:53.430762 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:07:53.430742 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd\": container with ID starting with 47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd not found: ID does not exist" containerID="47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd" Sep 17 19:07:53.430815 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430767 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd"} err="failed to get container status \"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd\": rpc error: code = NotFound desc = could not find container \"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd\": container with ID starting with 47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd not found: ID does not exist" Sep 17 19:07:53.430815 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.430784 2559 scope.go:117] "RemoveContainer" containerID="bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6" Sep 17 19:07:53.431046 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.431030 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6"} err="failed to get container status \"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6\": rpc error: code = NotFound desc = could not find container \"bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6\": container with ID starting with bc38cb6d76d0933cddc25bd76d060c5fa51da326846d658b69a650624a3a12d6 not found: ID does not exist" Sep 17 19:07:53.431101 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.431054 2559 scope.go:117] "RemoveContainer" containerID="96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249" Sep 17 19:07:53.431294 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.431279 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249"} err="failed to get container status \"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249\": rpc error: code = NotFound desc = could not find container \"96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249\": container with ID starting with 96969093cf814d5bb9337b5cf7831f5874ea16671e1a2fb64b9037bdd6091249 not found: ID does not exist" Sep 17 19:07:53.431333 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.431294 2559 scope.go:117] "RemoveContainer" containerID="47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd" Sep 17 19:07:53.431563 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.431533 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd"} err="failed to get container status \"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd\": rpc error: code = NotFound desc = could not find container \"47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd\": container with ID starting with 47998cfc7f8ca57faad4fffbaebcc92b7bf0f69664812e018554b67515d952dd not found: ID does not exist" Sep 17 19:07:53.449261 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.449233 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:07:53.453490 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.453468 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-69mtc"] Sep 17 19:07:53.992371 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:53.992340 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" path="/var/lib/kubelet/pods/d24ef56d-a6c4-411c-95af-678a42de1cdd/volumes" Sep 17 19:07:54.416711 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:54.416673 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:07:54.417960 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:54.417933 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:07:55.422374 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:55.422333 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:07:57.432798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:57.432759 2559 generic.go:356] "Generic (PLEG): container finished" podID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerID="0031a16587f55ea9184b9887b970563f6704cd14f6abfc770bf17ccb784bcd19" exitCode=0 Sep 17 19:07:57.433278 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:57.432812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerDied","Data":"0031a16587f55ea9184b9887b970563f6704cd14f6abfc770bf17ccb784bcd19"} Sep 17 19:07:58.439436 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:58.439399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerStarted","Data":"507b61cd3f6825a596d3facd0a40f1e2d92774674dffe575279706c10aec87f7"} Sep 17 19:07:58.439840 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:58.439475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerStarted","Data":"cb8aa77b43d83f56253c5d64d0c4b029cd5aafd792effb78d469309d3562cf65"} Sep 17 19:07:58.439840 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:58.439669 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:58.439840 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:58.439716 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:07:58.463140 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:07:58.463090 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" podStartSLOduration=4.756384898 podStartE2EDuration="8.463075043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.706690145s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:58.461099236 +0000 UTC m=+2330.449715486" watchObservedRunningTime="2026-09-17 19:07:58.463075043 +0000 UTC m=+2330.451691292" Sep 17 19:08:00.426416 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:00.426387 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:08:00.426918 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:00.426891 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:04.449531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:04.449498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:08:10.426889 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:10.426853 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:20.427404 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:20.427365 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:30.427068 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:30.427030 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:34.454167 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:34.454138 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:08:40.427119 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:40.427081 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:40.728664 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:40.728590 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:08:40.728888 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:40.728844 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kserve-container" containerID="cri-o://cb8aa77b43d83f56253c5d64d0c4b029cd5aafd792effb78d469309d3562cf65" gracePeriod=30 Sep 17 19:08:40.728963 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:40.728907 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kube-rbac-proxy" containerID="cri-o://507b61cd3f6825a596d3facd0a40f1e2d92774674dffe575279706c10aec87f7" gracePeriod=30 Sep 17 19:08:41.642891 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.642857 2559 generic.go:356] "Generic (PLEG): container finished" podID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerID="507b61cd3f6825a596d3facd0a40f1e2d92774674dffe575279706c10aec87f7" exitCode=2 Sep 17 19:08:41.642891 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.642885 2559 generic.go:356] "Generic (PLEG): container finished" podID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerID="cb8aa77b43d83f56253c5d64d0c4b029cd5aafd792effb78d469309d3562cf65" exitCode=0 Sep 17 19:08:41.643282 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.642924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerDied","Data":"507b61cd3f6825a596d3facd0a40f1e2d92774674dffe575279706c10aec87f7"} Sep 17 19:08:41.643282 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.642981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerDied","Data":"cb8aa77b43d83f56253c5d64d0c4b029cd5aafd792effb78d469309d3562cf65"} Sep 17 19:08:41.674545 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.674524 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:08:41.771829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.771800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") pod \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " Sep 17 19:08:41.771983 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.771863 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location\") pod \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " Sep 17 19:08:41.771983 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.771888 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsskq\" (UniqueName: \"kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq\") pod \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " Sep 17 19:08:41.771983 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.771929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert\") pod \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " Sep 17 19:08:41.771983 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.771968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config\") pod \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\" (UID: \"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd\") " Sep 17 19:08:41.772318 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.772229 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:08:41.772411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.772298 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:08:41.772411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.772385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:08:41.773896 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.773874 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:08:41.773996 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.773910 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq" (OuterVolumeSpecName: "kube-api-access-fsskq") pod "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" (UID: "c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd"). InnerVolumeSpecName "kube-api-access-fsskq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:08:41.873285 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.873261 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:08:41.873285 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.873281 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:08:41.873419 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.873291 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsskq\" (UniqueName: \"kubernetes.io/projected/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-api-access-fsskq\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:08:41.873419 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.873300 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:08:41.873419 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:41.873308 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:08:42.649198 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.649166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" event={"ID":"c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd","Type":"ContainerDied","Data":"09a85041b14ce9d09bdece91d5ae2cb1ac0e5836f7c75d9f4569a7fbab2f4e27"} Sep 17 19:08:42.649716 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.649216 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l" Sep 17 19:08:42.649716 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.649237 2559 scope.go:117] "RemoveContainer" containerID="507b61cd3f6825a596d3facd0a40f1e2d92774674dffe575279706c10aec87f7" Sep 17 19:08:42.657592 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.657576 2559 scope.go:117] "RemoveContainer" containerID="cb8aa77b43d83f56253c5d64d0c4b029cd5aafd792effb78d469309d3562cf65" Sep 17 19:08:42.667183 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.665440 2559 scope.go:117] "RemoveContainer" containerID="0031a16587f55ea9184b9887b970563f6704cd14f6abfc770bf17ccb784bcd19" Sep 17 19:08:42.669354 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.669334 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:08:42.672801 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:42.672778 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gcl2l"] Sep 17 19:08:43.992620 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:43.992585 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" path="/var/lib/kubelet/pods/c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd/volumes" Sep 17 19:08:45.974535 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974500 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974842 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974853 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974865 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974870 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974881 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="storage-initializer" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974888 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="storage-initializer" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974895 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kserve-container" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974901 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kserve-container" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974906 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kube-rbac-proxy" Sep 17 19:08:45.974907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974911 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kube-rbac-proxy" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974922 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="storage-initializer" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974927 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="storage-initializer" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974981 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kube-rbac-proxy" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974987 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kube-rbac-proxy" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.974995 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40b7e1d-da67-4cc9-b0c0-f1fa6684e7fd" containerName="kserve-container" Sep 17 19:08:45.975216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.975000 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24ef56d-a6c4-411c-95af-678a42de1cdd" containerName="kserve-container" Sep 17 19:08:45.977969 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.977951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:45.985181 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.985159 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\"" Sep 17 19:08:45.985335 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.985319 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\"" Sep 17 19:08:45.997208 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:45.997181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:08:46.109353 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.109317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.109353 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.109358 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.109612 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.109388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhw9\" (UniqueName: \"kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.109612 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.109503 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.109612 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.109573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhw9\" (UniqueName: \"kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.210953 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.210930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.211156 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.211135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.211222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.211167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.212912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.212882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.219626 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.219608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhw9\" (UniqueName: \"kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.289086 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.289028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:46.563744 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.563685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:08:46.565917 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:08:46.565885 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2678cce5_6849_4f33_a296_4b8c1cd17a6f.slice/crio-67fcf26bb931e3b7c020f1242dec4b066401da6b1fd868e4a41fc5e44450108c WatchSource:0}: Error finding container 67fcf26bb931e3b7c020f1242dec4b066401da6b1fd868e4a41fc5e44450108c: Status 404 returned error can't find the container with id 67fcf26bb931e3b7c020f1242dec4b066401da6b1fd868e4a41fc5e44450108c Sep 17 19:08:46.669240 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.669207 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerStarted","Data":"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7"} Sep 17 19:08:46.669398 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:46.669265 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerStarted","Data":"67fcf26bb931e3b7c020f1242dec4b066401da6b1fd868e4a41fc5e44450108c"} Sep 17 19:08:50.427363 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:50.427332 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:08:50.687936 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:50.687902 2559 generic.go:356] "Generic (PLEG): container finished" podID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerID="4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7" exitCode=0 Sep 17 19:08:50.688116 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:50.687974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerDied","Data":"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7"} Sep 17 19:08:51.694705 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.694670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerStarted","Data":"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7"} Sep 17 19:08:51.695125 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.694737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerStarted","Data":"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665"} Sep 17 19:08:51.695125 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.695025 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:51.695231 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.695173 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:51.696742 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.696718 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused" Sep 17 19:08:51.718442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:51.718392 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podStartSLOduration=2.9513600760000003 podStartE2EDuration="6.71837499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767014914s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:51.718051453 +0000 UTC m=+2383.706667714" watchObservedRunningTime="2026-09-17 19:08:51.71837499 +0000 UTC m=+2383.706991240" Sep 17 19:08:52.699942 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:52.699899 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused" Sep 17 19:08:57.704685 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:57.704652 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:08:57.705177 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:08:57.705151 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused" Sep 17 19:09:00.428200 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:00.428171 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:09:07.527048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.527017 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:09:07.527433 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.527253 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" containerID="cri-o://c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4" gracePeriod=30 Sep 17 19:09:07.527433 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.527277 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kube-rbac-proxy" containerID="cri-o://37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2" gracePeriod=30 Sep 17 19:09:07.705972 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.705945 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:09:07.773921 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.773887 2559 generic.go:356] "Generic (PLEG): container finished" podID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerID="37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2" exitCode=2 Sep 17 19:09:07.774061 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:07.773948 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerDied","Data":"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2"} Sep 17 19:09:10.093336 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.093307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:09:10.095197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.095173 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:09:10.100835 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.100816 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:09:10.102713 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.102688 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:09:10.423655 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.423611 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused" Sep 17 19:09:10.426919 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:10.426894 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused" Sep 17 19:09:11.361698 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.361676 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:09:11.403358 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403324 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert\") pod \"f911a896-c961-4700-b1bc-6b3a31d776ee\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " Sep 17 19:09:11.403550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403370 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls\") pod \"f911a896-c961-4700-b1bc-6b3a31d776ee\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " Sep 17 19:09:11.403550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location\") pod \"f911a896-c961-4700-b1bc-6b3a31d776ee\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " Sep 17 19:09:11.403550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403427 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config\") pod \"f911a896-c961-4700-b1bc-6b3a31d776ee\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " Sep 17 19:09:11.403550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403493 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fndtj\" (UniqueName: \"kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj\") pod \"f911a896-c961-4700-b1bc-6b3a31d776ee\" (UID: \"f911a896-c961-4700-b1bc-6b3a31d776ee\") " Sep 17 19:09:11.403786 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f911a896-c961-4700-b1bc-6b3a31d776ee" (UID: "f911a896-c961-4700-b1bc-6b3a31d776ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:09:11.403850 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403785 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f911a896-c961-4700-b1bc-6b3a31d776ee" (UID: "f911a896-c961-4700-b1bc-6b3a31d776ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:09:11.403850 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.403797 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f911a896-c961-4700-b1bc-6b3a31d776ee" (UID: "f911a896-c961-4700-b1bc-6b3a31d776ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:09:11.405579 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.405555 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj" (OuterVolumeSpecName: "kube-api-access-fndtj") pod "f911a896-c961-4700-b1bc-6b3a31d776ee" (UID: "f911a896-c961-4700-b1bc-6b3a31d776ee"). InnerVolumeSpecName "kube-api-access-fndtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:09:11.405579 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.405561 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f911a896-c961-4700-b1bc-6b3a31d776ee" (UID: "f911a896-c961-4700-b1bc-6b3a31d776ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:09:11.504798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.504770 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f911a896-c961-4700-b1bc-6b3a31d776ee-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:11.504798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.504796 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:11.504798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.504805 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndtj\" (UniqueName: \"kubernetes.io/projected/f911a896-c961-4700-b1bc-6b3a31d776ee-kube-api-access-fndtj\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:11.505004 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.504814 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f911a896-c961-4700-b1bc-6b3a31d776ee-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:11.505004 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.504822 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f911a896-c961-4700-b1bc-6b3a31d776ee-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:11.795114 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.795033 2559 generic.go:356] "Generic (PLEG): container finished" podID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerID="c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4" exitCode=0 Sep 17 19:09:11.795114 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.795079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerDied","Data":"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4"} Sep 17 19:09:11.795114 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.795100 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" Sep 17 19:09:11.795318 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.795132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6" event={"ID":"f911a896-c961-4700-b1bc-6b3a31d776ee","Type":"ContainerDied","Data":"d2c0a6822af50da4416b5e1b0679cf29e4bcc6566f57215af37947b5d027a4e8"} Sep 17 19:09:11.795318 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.795166 2559 scope.go:117] "RemoveContainer" containerID="37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2" Sep 17 19:09:11.804051 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.804031 2559 scope.go:117] "RemoveContainer" containerID="c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4" Sep 17 19:09:11.811327 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.811307 2559 scope.go:117] "RemoveContainer" containerID="4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2" Sep 17 19:09:11.818469 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.818423 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:09:11.818842 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.818827 2559 scope.go:117] "RemoveContainer" containerID="37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2" Sep 17 19:09:11.819110 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:11.819094 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2\": container with ID starting with 37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2 not found: ID does not exist" containerID="37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2" Sep 17 19:09:11.819166 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.819118 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2"} err="failed to get container status \"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2\": rpc error: code = NotFound desc = could not find container \"37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2\": container with ID starting with 37880e4dd6505aa690aa7452f6348200923caa1a4b9db02e7e858cb67e4bdda2 not found: ID does not exist" Sep 17 19:09:11.819166 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.819135 2559 scope.go:117] "RemoveContainer" containerID="c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4" Sep 17 19:09:11.819421 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:11.819403 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4\": container with ID starting with c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4 not found: ID does not exist" containerID="c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4" Sep 17 19:09:11.819564 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.819426 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4"} err="failed to get container status \"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4\": rpc error: code = NotFound desc = could not find container \"c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4\": container with ID starting with c837347ead54b09b577a2a51c123084198ff7d07f880fa75bfbb7564a77fe2e4 not found: ID does not exist" Sep 17 19:09:11.819564 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.819441 2559 scope.go:117] "RemoveContainer" containerID="4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2" Sep 17 19:09:11.819721 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:11.819703 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2\": container with ID starting with 4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2 not found: ID does not exist" containerID="4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2" Sep 17 19:09:11.819765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.819726 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2"} err="failed to get container status \"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2\": rpc error: code = NotFound desc = could not find container \"4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2\": container with ID starting with 4c373ec75366503b3a25568126cbe247d4e5c0b20dfaccc9665a0356fcace7f2 not found: ID does not exist" Sep 17 19:09:11.823321 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.823301 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-4hwx6"] Sep 17 19:09:11.992630 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:11.992609 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" path="/var/lib/kubelet/pods/f911a896-c961-4700-b1bc-6b3a31d776ee/volumes" Sep 17 19:09:12.682927 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.682895 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683203 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="storage-initializer" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683214 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="storage-initializer" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683226 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kube-rbac-proxy" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683232 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kube-rbac-proxy" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683243 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683248 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683299 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kube-rbac-proxy" Sep 17 19:09:12.683362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.683310 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f911a896-c961-4700-b1bc-6b3a31d776ee" containerName="kserve-container" Sep 17 19:09:12.686297 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.686281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.688998 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.688980 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\"" Sep 17 19:09:12.689222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.689209 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:09:12.697556 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.697536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:09:12.713836 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.713809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.713836 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.713845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.714045 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.713874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.714045 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.713931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knq7x\" (UniqueName: \"kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.714045 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.714006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.814810 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.814779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knq7x\" (UniqueName: \"kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.814965 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.814835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.814965 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.814871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.814965 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.814897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.815132 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.815013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.815424 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.815396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.815622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.815602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.815680 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.815647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.817431 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.817413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.828043 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.824738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knq7x\" (UniqueName: \"kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:12.995987 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:12.995921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:13.271356 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:13.271332 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:09:13.273395 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:09:13.273369 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ec3f7f_bd49_4d29_830e_8fed4c5c8cb9.slice/crio-a3bd714a27d44065c641409c75b3cfdfa55f3cd16b200c54b9acf0f8333aa1a9 WatchSource:0}: Error finding container a3bd714a27d44065c641409c75b3cfdfa55f3cd16b200c54b9acf0f8333aa1a9: Status 404 returned error can't find the container with id a3bd714a27d44065c641409c75b3cfdfa55f3cd16b200c54b9acf0f8333aa1a9 Sep 17 19:09:13.807531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:13.807486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerStarted","Data":"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049"} Sep 17 19:09:13.807912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:13.807578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerStarted","Data":"a3bd714a27d44065c641409c75b3cfdfa55f3cd16b200c54b9acf0f8333aa1a9"} Sep 17 19:09:16.024375 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.024345 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:09:16.024795 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.024579 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" containerID="cri-o://ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" gracePeriod=30 Sep 17 19:09:16.024795 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.024621 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kube-rbac-proxy" containerID="cri-o://42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" gracePeriod=30 Sep 17 19:09:16.761712 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.761688 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:09:16.822393 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822356 2559 generic.go:356] "Generic (PLEG): container finished" podID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerID="42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" exitCode=2 Sep 17 19:09:16.822393 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822381 2559 generic.go:356] "Generic (PLEG): container finished" podID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerID="ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" exitCode=0 Sep 17 19:09:16.822619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822408 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" Sep 17 19:09:16.822619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerDied","Data":"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7"} Sep 17 19:09:16.822619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerDied","Data":"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665"} Sep 17 19:09:16.822619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz" event={"ID":"2678cce5-6849-4f33-a296-4b8c1cd17a6f","Type":"ContainerDied","Data":"67fcf26bb931e3b7c020f1242dec4b066401da6b1fd868e4a41fc5e44450108c"} Sep 17 19:09:16.822619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.822548 2559 scope.go:117] "RemoveContainer" containerID="42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" Sep 17 19:09:16.830598 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.830577 2559 scope.go:117] "RemoveContainer" containerID="ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" Sep 17 19:09:16.838318 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.838298 2559 scope.go:117] "RemoveContainer" containerID="4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7" Sep 17 19:09:16.846097 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.846079 2559 scope.go:117] "RemoveContainer" containerID="42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" Sep 17 19:09:16.846400 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:16.846384 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7\": container with ID starting with 42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7 not found: ID does not exist" containerID="42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" Sep 17 19:09:16.846482 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.846407 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7"} err="failed to get container status \"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7\": rpc error: code = NotFound desc = could not find container \"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7\": container with ID starting with 42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7 not found: ID does not exist" Sep 17 19:09:16.846482 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.846422 2559 scope.go:117] "RemoveContainer" containerID="ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" Sep 17 19:09:16.846726 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:16.846708 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665\": container with ID starting with ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665 not found: ID does not exist" containerID="ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" Sep 17 19:09:16.846769 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.846735 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665"} err="failed to get container status \"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665\": rpc error: code = NotFound desc = could not find container \"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665\": container with ID starting with ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665 not found: ID does not exist" Sep 17 19:09:16.846769 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.846751 2559 scope.go:117] "RemoveContainer" containerID="4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7" Sep 17 19:09:16.847028 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:16.847005 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7\": container with ID starting with 4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7 not found: ID does not exist" containerID="4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7" Sep 17 19:09:16.847074 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847037 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7"} err="failed to get container status \"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7\": rpc error: code = NotFound desc = could not find container \"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7\": container with ID starting with 4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7 not found: ID does not exist" Sep 17 19:09:16.847074 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847059 2559 scope.go:117] "RemoveContainer" containerID="42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7" Sep 17 19:09:16.847317 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847298 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7"} err="failed to get container status \"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7\": rpc error: code = NotFound desc = could not find container \"42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7\": container with ID starting with 42516b376cde3ef6d86c81ba4e9ce662f1223620bf47c73597b3439f01e3ebf7 not found: ID does not exist" Sep 17 19:09:16.847317 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847318 2559 scope.go:117] "RemoveContainer" containerID="ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665" Sep 17 19:09:16.847577 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847551 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665"} err="failed to get container status \"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665\": rpc error: code = NotFound desc = could not find container \"ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665\": container with ID starting with ef1a52c659b89b633dff6191c40d1b7878e987ef768514211d24f4318d7f6665 not found: ID does not exist" Sep 17 19:09:16.847577 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847578 2559 scope.go:117] "RemoveContainer" containerID="4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7" Sep 17 19:09:16.847858 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.847840 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7"} err="failed to get container status \"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7\": rpc error: code = NotFound desc = could not find container \"4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7\": container with ID starting with 4b5f648e808f1238d68cd676ae2968d4bea40e7bf7d95edc6f7676cc0a61b4a7 not found: ID does not exist" Sep 17 19:09:16.850116 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850098 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls\") pod \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " Sep 17 19:09:16.850177 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850133 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert\") pod \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " Sep 17 19:09:16.850177 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location\") pod \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " Sep 17 19:09:16.850286 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config\") pod \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " Sep 17 19:09:16.850286 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850258 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvhw9\" (UniqueName: \"kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9\") pod \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\" (UID: \"2678cce5-6849-4f33-a296-4b8c1cd17a6f\") " Sep 17 19:09:16.850540 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850515 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2678cce5-6849-4f33-a296-4b8c1cd17a6f" (UID: "2678cce5-6849-4f33-a296-4b8c1cd17a6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:09:16.850650 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850548 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2678cce5-6849-4f33-a296-4b8c1cd17a6f" (UID: "2678cce5-6849-4f33-a296-4b8c1cd17a6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:09:16.850702 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.850647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2678cce5-6849-4f33-a296-4b8c1cd17a6f" (UID: "2678cce5-6849-4f33-a296-4b8c1cd17a6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:09:16.852284 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.852254 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9" (OuterVolumeSpecName: "kube-api-access-xvhw9") pod "2678cce5-6849-4f33-a296-4b8c1cd17a6f" (UID: "2678cce5-6849-4f33-a296-4b8c1cd17a6f"). InnerVolumeSpecName "kube-api-access-xvhw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:09:16.852356 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.852282 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2678cce5-6849-4f33-a296-4b8c1cd17a6f" (UID: "2678cce5-6849-4f33-a296-4b8c1cd17a6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:09:16.951509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.951476 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:16.951509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.951503 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhw9\" (UniqueName: \"kubernetes.io/projected/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kube-api-access-xvhw9\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:16.951509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.951512 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2678cce5-6849-4f33-a296-4b8c1cd17a6f-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:16.951707 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.951521 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2678cce5-6849-4f33-a296-4b8c1cd17a6f-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:16.951707 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:16.951531 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2678cce5-6849-4f33-a296-4b8c1cd17a6f-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:09:17.155402 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:17.155371 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:09:17.162995 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:17.162971 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r5qpz"] Sep 17 19:09:17.827870 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:17.827836 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerID="ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049" exitCode=0 Sep 17 19:09:17.828062 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:17.827911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerDied","Data":"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049"} Sep 17 19:09:17.992287 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:17.992257 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" path="/var/lib/kubelet/pods/2678cce5-6849-4f33-a296-4b8c1cd17a6f/volumes" Sep 17 19:09:18.835375 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:18.835343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerStarted","Data":"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b"} Sep 17 19:09:18.835809 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:18.835402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerStarted","Data":"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632"} Sep 17 19:09:18.835809 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:18.835606 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:18.862388 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:18.862341 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" podStartSLOduration=3.296177085 podStartE2EDuration="6.862326919s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.566149834s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:18.861589239 +0000 UTC m=+2410.850205487" watchObservedRunningTime="2026-09-17 19:09:18.862326919 +0000 UTC m=+2410.850943235" Sep 17 19:09:19.839315 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:19.839280 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:21.367166 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367064 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367410 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="storage-initializer" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367421 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="storage-initializer" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367433 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kube-rbac-proxy" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367438 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kube-rbac-proxy" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367463 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367470 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367524 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kube-rbac-proxy" Sep 17 19:09:21.367550 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.367533 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2678cce5-6849-4f33-a296-4b8c1cd17a6f" containerName="kserve-container" Sep 17 19:09:21.369736 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.369721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.372740 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.372719 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\"" Sep 17 19:09:21.373081 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.373068 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:09:21.400924 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.400899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:09:21.495355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.495333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.495500 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.495376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.495500 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.495395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.495613 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.495494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.495613 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.495530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przpc\" (UniqueName: \"kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przpc\" (UniqueName: \"kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.596731 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:21.596661 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found Sep 17 19:09:21.596920 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:09:21.596745 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls podName:5b97643a-84c0-4936-9b60-b5f91f47594b nodeName:}" failed. No retries permitted until 2026-09-17 19:09:22.096722697 +0000 UTC m=+2414.085338937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found Sep 17 19:09:21.596969 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.596955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.597365 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.597342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.597404 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.597371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:21.610103 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:21.610073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-przpc\" (UniqueName: \"kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:22.099720 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.099688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:22.101929 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.101902 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:22.279156 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.279122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:22.573876 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.573840 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:09:22.576325 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:09:22.576290 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b97643a_84c0_4936_9b60_b5f91f47594b.slice/crio-a2b3834e05ead71e19038e7ef33d152cd644513b81820dc4babf31aec184f00a WatchSource:0}: Error finding container a2b3834e05ead71e19038e7ef33d152cd644513b81820dc4babf31aec184f00a: Status 404 returned error can't find the container with id a2b3834e05ead71e19038e7ef33d152cd644513b81820dc4babf31aec184f00a Sep 17 19:09:22.854244 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.854141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerStarted","Data":"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80"} Sep 17 19:09:22.854244 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:22.854224 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerStarted","Data":"a2b3834e05ead71e19038e7ef33d152cd644513b81820dc4babf31aec184f00a"} Sep 17 19:09:25.849430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:25.849397 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:09:26.874013 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:26.873939 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerID="bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80" exitCode=0 Sep 17 19:09:26.874349 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:26.874010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerDied","Data":"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80"} Sep 17 19:09:27.882129 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:27.882095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerStarted","Data":"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e"} Sep 17 19:09:27.882560 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:27.882156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerStarted","Data":"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52"} Sep 17 19:09:27.882560 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:27.882368 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:27.882560 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:27.882426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:27.909989 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:27.909934 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" podStartSLOduration=3.08012817 podStartE2EDuration="6.909917685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829789515s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:27.908297135 +0000 UTC m=+2419.896913384" watchObservedRunningTime="2026-09-17 19:09:27.909917685 +0000 UTC m=+2419.898533934" Sep 17 19:09:33.892443 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:33.892411 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:09:55.854081 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:09:55.854032 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Sep 17 19:10:03.897185 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:03.897151 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:10:05.852785 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:05.852753 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:10:11.290152 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:11.290115 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:10:11.291962 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:11.290425 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kserve-container" containerID="cri-o://62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52" gracePeriod=30 Sep 17 19:10:11.291962 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:11.290495 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kube-rbac-proxy" containerID="cri-o://c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e" gracePeriod=30 Sep 17 19:10:12.087046 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.087011 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerID="c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e" exitCode=2 Sep 17 19:10:12.087222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.087078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerDied","Data":"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e"} Sep 17 19:10:12.533975 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.533951 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:10:12.607350 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607320 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert\") pod \"5b97643a-84c0-4936-9b60-b5f91f47594b\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " Sep 17 19:10:12.607530 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-przpc\" (UniqueName: \"kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc\") pod \"5b97643a-84c0-4936-9b60-b5f91f47594b\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " Sep 17 19:10:12.607530 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") pod \"5b97643a-84c0-4936-9b60-b5f91f47594b\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " Sep 17 19:10:12.607530 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607513 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config\") pod \"5b97643a-84c0-4936-9b60-b5f91f47594b\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " Sep 17 19:10:12.607715 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607567 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location\") pod \"5b97643a-84c0-4936-9b60-b5f91f47594b\" (UID: \"5b97643a-84c0-4936-9b60-b5f91f47594b\") " Sep 17 19:10:12.607869 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607849 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b97643a-84c0-4936-9b60-b5f91f47594b" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:10:12.607920 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607901 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b97643a-84c0-4936-9b60-b5f91f47594b" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:10:12.607997 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.607971 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b97643a-84c0-4936-9b60-b5f91f47594b" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:10:12.609561 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.609538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b97643a-84c0-4936-9b60-b5f91f47594b" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:10:12.609631 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.609600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc" (OuterVolumeSpecName: "kube-api-access-przpc") pod "5b97643a-84c0-4936-9b60-b5f91f47594b" (UID: "5b97643a-84c0-4936-9b60-b5f91f47594b"). InnerVolumeSpecName "kube-api-access-przpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:10:12.709190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.709101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:12.709190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.709141 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b97643a-84c0-4936-9b60-b5f91f47594b-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:12.709190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.709152 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b97643a-84c0-4936-9b60-b5f91f47594b-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:12.709190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.709161 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przpc\" (UniqueName: \"kubernetes.io/projected/5b97643a-84c0-4936-9b60-b5f91f47594b-kube-api-access-przpc\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:12.709190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.709171 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b97643a-84c0-4936-9b60-b5f91f47594b-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:12.876998 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.876964 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:10:12.877238 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.877194 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kserve-container" containerID="cri-o://ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632" gracePeriod=30 Sep 17 19:10:12.877322 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:12.877238 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kube-rbac-proxy" containerID="cri-o://263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b" gracePeriod=30 Sep 17 19:10:13.094653 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.094616 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerID="263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b" exitCode=2 Sep 17 19:10:13.094829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.094684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerDied","Data":"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b"} Sep 17 19:10:13.096624 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.096604 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerID="62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52" exitCode=0 Sep 17 19:10:13.096715 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.096662 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" Sep 17 19:10:13.096715 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.096683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerDied","Data":"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52"} Sep 17 19:10:13.096794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.096739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq" event={"ID":"5b97643a-84c0-4936-9b60-b5f91f47594b","Type":"ContainerDied","Data":"a2b3834e05ead71e19038e7ef33d152cd644513b81820dc4babf31aec184f00a"} Sep 17 19:10:13.096794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.096767 2559 scope.go:117] "RemoveContainer" containerID="c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e" Sep 17 19:10:13.106187 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.106171 2559 scope.go:117] "RemoveContainer" containerID="62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52" Sep 17 19:10:13.114156 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.114138 2559 scope.go:117] "RemoveContainer" containerID="bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80" Sep 17 19:10:13.122347 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122325 2559 scope.go:117] "RemoveContainer" containerID="c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e" Sep 17 19:10:13.122432 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122417 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:10:13.122659 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:13.122638 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e\": container with ID starting with c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e not found: ID does not exist" containerID="c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e" Sep 17 19:10:13.122710 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122664 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e"} err="failed to get container status \"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e\": rpc error: code = NotFound desc = could not find container \"c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e\": container with ID starting with c1fa43a2c1d2ad05a7b82523498d8c21550053b7c1741ec3bbe66f440765453e not found: ID does not exist" Sep 17 19:10:13.122710 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122681 2559 scope.go:117] "RemoveContainer" containerID="62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52" Sep 17 19:10:13.122960 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:13.122941 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52\": container with ID starting with 62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52 not found: ID does not exist" containerID="62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52" Sep 17 19:10:13.123013 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122968 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52"} err="failed to get container status \"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52\": rpc error: code = NotFound desc = could not find container \"62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52\": container with ID starting with 62252ba7ab73276c00571e7b747c376f0e3487b75fe0a4b8524901d7a105fb52 not found: ID does not exist" Sep 17 19:10:13.123013 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.122983 2559 scope.go:117] "RemoveContainer" containerID="bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80" Sep 17 19:10:13.123201 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:13.123186 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80\": container with ID starting with bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80 not found: ID does not exist" containerID="bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80" Sep 17 19:10:13.123240 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.123203 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80"} err="failed to get container status \"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80\": rpc error: code = NotFound desc = could not find container \"bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80\": container with ID starting with bc13b413db07e44f3374cb3d300e7aa357b40b1dc7000f763de549be7c00aa80 not found: ID does not exist" Sep 17 19:10:13.127705 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.127685 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-rz2bq"] Sep 17 19:10:13.991989 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:13.991958 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" path="/var/lib/kubelet/pods/5b97643a-84c0-4936-9b60-b5f91f47594b/volumes" Sep 17 19:10:15.843885 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:15.843841 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused" Sep 17 19:10:16.565948 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.565909 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:10:16.566424 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566405 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kube-rbac-proxy" Sep 17 19:10:16.566480 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566428 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kube-rbac-proxy" Sep 17 19:10:16.566480 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566465 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kserve-container" Sep 17 19:10:16.566480 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566475 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kserve-container" Sep 17 19:10:16.566622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566494 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="storage-initializer" Sep 17 19:10:16.566622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566504 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="storage-initializer" Sep 17 19:10:16.566622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566584 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kserve-container" Sep 17 19:10:16.566622 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.566601 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b97643a-84c0-4936-9b60-b5f91f47594b" containerName="kube-rbac-proxy" Sep 17 19:10:16.570417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.570398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.573167 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.573143 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\"" Sep 17 19:10:16.573396 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.573381 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\"" Sep 17 19:10:16.592869 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.592837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:10:16.641094 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.641056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.641254 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.641096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.641254 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.641164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.641357 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.641251 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxfqf\" (UniqueName: \"kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.641357 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.641331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.741997 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.741957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxfqf\" (UniqueName: \"kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.741997 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742229 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742229 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742229 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742555 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742787 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.742829 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.742797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.744653 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.744625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.758679 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.758656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxfqf\" (UniqueName: \"kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf\") pod \"isvc-sklearn-mcp-predictor-7bdb858467-574f2\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:16.880569 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:16.880480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:17.184104 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:17.184077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:10:17.185780 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:10:17.185753 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eeb9d99_567f_4a90_b5b0_adb7f68d0c9e.slice/crio-6c35f84860cda42841cb4bbbf98cd3616d287b620f8fb9541133f30a568c8f81 WatchSource:0}: Error finding container 6c35f84860cda42841cb4bbbf98cd3616d287b620f8fb9541133f30a568c8f81: Status 404 returned error can't find the container with id 6c35f84860cda42841cb4bbbf98cd3616d287b620f8fb9541133f30a568c8f81 Sep 17 19:10:18.122681 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:18.122644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerStarted","Data":"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae"} Sep 17 19:10:18.123049 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:18.122709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerStarted","Data":"6c35f84860cda42841cb4bbbf98cd3616d287b620f8fb9541133f30a568c8f81"} Sep 17 19:10:20.510799 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.510774 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:10:20.574634 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.574605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls\") pod \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " Sep 17 19:10:20.574828 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.574668 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config\") pod \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " Sep 17 19:10:20.574828 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.574718 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location\") pod \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " Sep 17 19:10:20.574828 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.574778 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knq7x\" (UniqueName: \"kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x\") pod \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " Sep 17 19:10:20.574828 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.574800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert\") pod \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\" (UID: \"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9\") " Sep 17 19:10:20.575091 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.575068 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" (UID: "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:10:20.575136 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.575118 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" (UID: "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:10:20.575208 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.575181 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" (UID: "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:10:20.576824 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.576794 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" (UID: "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:10:20.576913 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.576832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x" (OuterVolumeSpecName: "kube-api-access-knq7x") pod "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" (UID: "b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9"). InnerVolumeSpecName "kube-api-access-knq7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:10:20.676105 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.676070 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:20.676105 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.676101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:20.676105 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.676113 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:20.676359 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.676122 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knq7x\" (UniqueName: \"kubernetes.io/projected/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-kube-api-access-knq7x\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:20.676359 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:20.676131 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:10:21.136541 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.136513 2559 generic.go:356] "Generic (PLEG): container finished" podID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerID="760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae" exitCode=0 Sep 17 19:10:21.136683 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.136582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerDied","Data":"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae"} Sep 17 19:10:21.138636 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.138613 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerID="ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632" exitCode=0 Sep 17 19:10:21.138751 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.138651 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerDied","Data":"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632"} Sep 17 19:10:21.138751 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.138665 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" Sep 17 19:10:21.138751 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.138691 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p" event={"ID":"b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9","Type":"ContainerDied","Data":"a3bd714a27d44065c641409c75b3cfdfa55f3cd16b200c54b9acf0f8333aa1a9"} Sep 17 19:10:21.138751 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.138722 2559 scope.go:117] "RemoveContainer" containerID="263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b" Sep 17 19:10:21.149845 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.149829 2559 scope.go:117] "RemoveContainer" containerID="ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632" Sep 17 19:10:21.158167 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.158151 2559 scope.go:117] "RemoveContainer" containerID="ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049" Sep 17 19:10:21.172880 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.172859 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:10:21.177347 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.177259 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kjr7p"] Sep 17 19:10:21.177416 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.177370 2559 scope.go:117] "RemoveContainer" containerID="263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b" Sep 17 19:10:21.177694 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:21.177672 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b\": container with ID starting with 263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b not found: ID does not exist" containerID="263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b" Sep 17 19:10:21.177783 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.177704 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b"} err="failed to get container status \"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b\": rpc error: code = NotFound desc = could not find container \"263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b\": container with ID starting with 263d7e8962b313922b8cc37c2c57030bfe921f415014b51eb2370803c9dea97b not found: ID does not exist" Sep 17 19:10:21.177783 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.177731 2559 scope.go:117] "RemoveContainer" containerID="ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632" Sep 17 19:10:21.178062 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:21.178031 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632\": container with ID starting with ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632 not found: ID does not exist" containerID="ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632" Sep 17 19:10:21.178165 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.178067 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632"} err="failed to get container status \"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632\": rpc error: code = NotFound desc = could not find container \"ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632\": container with ID starting with ab6a1a7190519212e303a76d3648de2173786eec5f84abf944026b31a9338632 not found: ID does not exist" Sep 17 19:10:21.178165 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.178083 2559 scope.go:117] "RemoveContainer" containerID="ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049" Sep 17 19:10:21.178359 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:10:21.178326 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049\": container with ID starting with ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049 not found: ID does not exist" containerID="ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049" Sep 17 19:10:21.178359 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.178351 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049"} err="failed to get container status \"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049\": rpc error: code = NotFound desc = could not find container \"ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049\": container with ID starting with ba529af57783c44b3dbcebbcfd620a12c023c0991eb87314d5492d4d1a7f7049 not found: ID does not exist" Sep 17 19:10:21.995806 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:21.995771 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" path="/var/lib/kubelet/pods/b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9/volumes" Sep 17 19:10:22.146989 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:22.146951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerStarted","Data":"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede"} Sep 17 19:10:24.161840 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:24.161803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerStarted","Data":"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe"} Sep 17 19:10:24.162320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:24.161867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerStarted","Data":"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f"} Sep 17 19:10:24.162320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:24.161980 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:24.162320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:24.162012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:24.186638 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:24.186538 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podStartSLOduration=1.632409679 podStartE2EDuration="8.186523872s" totalImagesPullingTime="2.726032795s" totalInitContainerRuntime="3.828081398s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:24.184086362 +0000 UTC m=+2476.172702611" watchObservedRunningTime="2026-09-17 19:10:24.186523872 +0000 UTC m=+2476.175140121" Sep 17 19:10:25.167041 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:25.167001 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:31.177723 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:31.177692 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:10:51.179232 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:51.179193 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused" Sep 17 19:10:56.881705 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:10:56.881659 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused" Sep 17 19:11:01.181629 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:01.181593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:11:31.182583 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:31.182546 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:11:36.618191 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:36.618158 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:11:36.618601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:36.618425 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" containerID="cri-o://d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede" gracePeriod=30 Sep 17 19:11:36.618601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:36.618494 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" containerID="cri-o://fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe" gracePeriod=30 Sep 17 19:11:36.618725 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:36.618523 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-agent" containerID="cri-o://0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f" gracePeriod=30 Sep 17 19:11:37.470396 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:37.470354 2559 generic.go:356] "Generic (PLEG): container finished" podID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerID="fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe" exitCode=2 Sep 17 19:11:37.470582 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:37.470429 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerDied","Data":"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe"} Sep 17 19:11:39.480792 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:39.480759 2559 generic.go:356] "Generic (PLEG): container finished" podID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerID="d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede" exitCode=0 Sep 17 19:11:39.481219 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:39.480794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerDied","Data":"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede"} Sep 17 19:11:41.172934 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.172890 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:11:41.178373 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.178345 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused" Sep 17 19:11:41.767092 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767054 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:11:41.767416 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767404 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kube-rbac-proxy" Sep 17 19:11:41.767486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767417 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kube-rbac-proxy" Sep 17 19:11:41.767486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767426 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="storage-initializer" Sep 17 19:11:41.767486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767431 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="storage-initializer" Sep 17 19:11:41.767486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767439 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kserve-container" Sep 17 19:11:41.767486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767458 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kserve-container" Sep 17 19:11:41.767651 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767503 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kserve-container" Sep 17 19:11:41.767651 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.767514 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ec3f7f-bd49-4d29-830e-8fed4c5c8cb9" containerName="kube-rbac-proxy" Sep 17 19:11:41.770085 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.770069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.772643 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.772616 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\"" Sep 17 19:11:41.772778 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.772615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\"" Sep 17 19:11:41.782145 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.782122 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:11:41.837353 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.837322 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.837508 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.837367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.837508 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.837468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxkf\" (UniqueName: \"kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.837636 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.837520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.837636 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.837568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.938849 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.938817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.938863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.938993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939134 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.939058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939134 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.939115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxkf\" (UniqueName: \"kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939374 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.939351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939604 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.939585 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.939728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.939709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.941379 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.941358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:41.947790 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:41.947769 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxkf\" (UniqueName: \"kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-dfwx6\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:42.079523 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:42.079457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:42.351551 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:42.351527 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:11:42.354079 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:11:42.354050 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057de55c_8402_4ca5_99d5_858100ed7520.slice/crio-d24f73ae9fc282784d6cca1b93694ee485a3420ffb2e260eed37943538b48e7c WatchSource:0}: Error finding container d24f73ae9fc282784d6cca1b93694ee485a3420ffb2e260eed37943538b48e7c: Status 404 returned error can't find the container with id d24f73ae9fc282784d6cca1b93694ee485a3420ffb2e260eed37943538b48e7c Sep 17 19:11:42.494401 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:42.494368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerStarted","Data":"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08"} Sep 17 19:11:42.494561 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:42.494425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerStarted","Data":"d24f73ae9fc282784d6cca1b93694ee485a3420ffb2e260eed37943538b48e7c"} Sep 17 19:11:46.172613 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:46.172579 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:11:47.516701 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:47.516664 2559 generic.go:356] "Generic (PLEG): container finished" podID="057de55c-8402-4ca5-99d5-858100ed7520" containerID="b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08" exitCode=0 Sep 17 19:11:47.516701 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:47.516703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerDied","Data":"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08"} Sep 17 19:11:51.172378 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:51.172332 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:11:51.172855 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:51.172538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:11:51.178945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:51.178915 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused" Sep 17 19:11:56.173151 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:56.173096 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:11:59.575872 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:59.575839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerStarted","Data":"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631"} Sep 17 19:11:59.576309 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:59.575906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerStarted","Data":"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da"} Sep 17 19:11:59.576309 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:59.576092 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:11:59.595289 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:11:59.595230 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podStartSLOduration=2.650511912 podStartE2EDuration="18.595216384s" totalImagesPullingTime="11.543445154s" totalInitContainerRuntime="4.401259318s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:59.594658598 +0000 UTC m=+2571.583274851" watchObservedRunningTime="2026-09-17 19:11:59.595216384 +0000 UTC m=+2571.583832634" Sep 17 19:12:00.580650 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:00.580614 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:12:00.581957 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:00.581931 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:01.173179 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:01.173135 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:12:01.178617 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:01.178589 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused" Sep 17 19:12:01.178715 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:01.178701 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:12:01.585097 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:01.585002 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:06.173067 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.173027 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused" Sep 17 19:12:06.589159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.589090 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:12:06.589610 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.589583 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:06.807235 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.807211 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:12:06.941683 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.941649 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxfqf\" (UniqueName: \"kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf\") pod \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " Sep 17 19:12:06.941864 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.941690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config\") pod \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " Sep 17 19:12:06.941864 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.941723 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location\") pod \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " Sep 17 19:12:06.941864 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.941780 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls\") pod \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " Sep 17 19:12:06.941864 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.941824 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert\") pod \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\" (UID: \"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e\") " Sep 17 19:12:06.942160 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.942086 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" (UID: "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:12:06.942273 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.942239 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" (UID: "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:12:06.942273 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.942263 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" (UID: "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:12:06.944411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.944391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf" (OuterVolumeSpecName: "kube-api-access-nxfqf") pod "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" (UID: "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e"). InnerVolumeSpecName "kube-api-access-nxfqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:12:06.944411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:06.944400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" (UID: "8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:12:07.042649 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.042607 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:07.042649 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.042644 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:07.042862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.042660 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxfqf\" (UniqueName: \"kubernetes.io/projected/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-api-access-nxfqf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:07.042862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.042674 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:07.042862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.042688 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:07.611194 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.611157 2559 generic.go:356] "Generic (PLEG): container finished" podID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerID="0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f" exitCode=137 Sep 17 19:12:07.611601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.611226 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" Sep 17 19:12:07.611601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.611229 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerDied","Data":"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f"} Sep 17 19:12:07.611601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.611289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2" event={"ID":"8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e","Type":"ContainerDied","Data":"6c35f84860cda42841cb4bbbf98cd3616d287b620f8fb9541133f30a568c8f81"} Sep 17 19:12:07.611601 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.611321 2559 scope.go:117] "RemoveContainer" containerID="fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe" Sep 17 19:12:07.619884 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.619868 2559 scope.go:117] "RemoveContainer" containerID="0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f" Sep 17 19:12:07.627368 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.627223 2559 scope.go:117] "RemoveContainer" containerID="d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede" Sep 17 19:12:07.634544 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.634480 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:12:07.634593 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.634560 2559 scope.go:117] "RemoveContainer" containerID="760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae" Sep 17 19:12:07.640136 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.640117 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7bdb858467-574f2"] Sep 17 19:12:07.642032 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642016 2559 scope.go:117] "RemoveContainer" containerID="fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe" Sep 17 19:12:07.642288 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:07.642274 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe\": container with ID starting with fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe not found: ID does not exist" containerID="fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe" Sep 17 19:12:07.642334 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642300 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe"} err="failed to get container status \"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe\": rpc error: code = NotFound desc = could not find container \"fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe\": container with ID starting with fc042393afd172dce4914235109b1ee4e6f251d173b3946b73512046fc73fdbe not found: ID does not exist" Sep 17 19:12:07.642334 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642315 2559 scope.go:117] "RemoveContainer" containerID="0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f" Sep 17 19:12:07.642560 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:07.642544 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f\": container with ID starting with 0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f not found: ID does not exist" containerID="0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f" Sep 17 19:12:07.642618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642564 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f"} err="failed to get container status \"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f\": rpc error: code = NotFound desc = could not find container \"0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f\": container with ID starting with 0a2df1816eef8e809132def24f559f4a2d8eee9b082b4fb05ec95d694cdd868f not found: ID does not exist" Sep 17 19:12:07.642618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642586 2559 scope.go:117] "RemoveContainer" containerID="d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede" Sep 17 19:12:07.642828 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:07.642785 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede\": container with ID starting with d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede not found: ID does not exist" containerID="d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede" Sep 17 19:12:07.642888 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642837 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede"} err="failed to get container status \"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede\": rpc error: code = NotFound desc = could not find container \"d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede\": container with ID starting with d496e42d0db4fd130e1750eb58b489b3c1fd181b12a41a309ba4c486f19e3ede not found: ID does not exist" Sep 17 19:12:07.642888 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.642858 2559 scope.go:117] "RemoveContainer" containerID="760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae" Sep 17 19:12:07.643089 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:07.643073 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae\": container with ID starting with 760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae not found: ID does not exist" containerID="760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae" Sep 17 19:12:07.643131 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.643091 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae"} err="failed to get container status \"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae\": rpc error: code = NotFound desc = could not find container \"760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae\": container with ID starting with 760ce7ef090b4a30dd9bbd39ed4028dd1e87d806bdb47d2cedb6edb1f5d729ae not found: ID does not exist" Sep 17 19:12:07.992935 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:07.992902 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" path="/var/lib/kubelet/pods/8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e/volumes" Sep 17 19:12:16.589625 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:16.589579 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:26.590124 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:26.590040 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:36.589898 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:36.589857 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused" Sep 17 19:12:46.591104 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:46.591068 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:12:53.187258 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:53.187224 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:12:53.187770 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:53.187475 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" containerID="cri-o://d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da" gracePeriod=30 Sep 17 19:12:53.187770 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:53.187535 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kube-rbac-proxy" containerID="cri-o://2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631" gracePeriod=30 Sep 17 19:12:53.803465 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:53.803413 2559 generic.go:356] "Generic (PLEG): container finished" podID="057de55c-8402-4ca5-99d5-858100ed7520" containerID="2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631" exitCode=2 Sep 17 19:12:53.803635 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:53.803471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerDied","Data":"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631"} Sep 17 19:12:55.728828 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.728807 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:12:55.813493 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.813441 2559 generic.go:356] "Generic (PLEG): container finished" podID="057de55c-8402-4ca5-99d5-858100ed7520" containerID="d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da" exitCode=0 Sep 17 19:12:55.813617 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.813485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerDied","Data":"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da"} Sep 17 19:12:55.813617 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.813514 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" Sep 17 19:12:55.813617 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.813543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6" event={"ID":"057de55c-8402-4ca5-99d5-858100ed7520","Type":"ContainerDied","Data":"d24f73ae9fc282784d6cca1b93694ee485a3420ffb2e260eed37943538b48e7c"} Sep 17 19:12:55.813617 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.813571 2559 scope.go:117] "RemoveContainer" containerID="2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631" Sep 17 19:12:55.818314 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818258 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config\") pod \"057de55c-8402-4ca5-99d5-858100ed7520\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " Sep 17 19:12:55.818314 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818305 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location\") pod \"057de55c-8402-4ca5-99d5-858100ed7520\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " Sep 17 19:12:55.818442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818334 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwxkf\" (UniqueName: \"kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf\") pod \"057de55c-8402-4ca5-99d5-858100ed7520\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " Sep 17 19:12:55.818442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818399 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert\") pod \"057de55c-8402-4ca5-99d5-858100ed7520\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " Sep 17 19:12:55.818442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818425 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls\") pod \"057de55c-8402-4ca5-99d5-858100ed7520\" (UID: \"057de55c-8402-4ca5-99d5-858100ed7520\") " Sep 17 19:12:55.818768 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818741 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "057de55c-8402-4ca5-99d5-858100ed7520" (UID: "057de55c-8402-4ca5-99d5-858100ed7520"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:12:55.818862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.818781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "057de55c-8402-4ca5-99d5-858100ed7520" (UID: "057de55c-8402-4ca5-99d5-858100ed7520"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:12:55.820460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.820421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "057de55c-8402-4ca5-99d5-858100ed7520" (UID: "057de55c-8402-4ca5-99d5-858100ed7520"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:12:55.820607 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.820578 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf" (OuterVolumeSpecName: "kube-api-access-gwxkf") pod "057de55c-8402-4ca5-99d5-858100ed7520" (UID: "057de55c-8402-4ca5-99d5-858100ed7520"). InnerVolumeSpecName "kube-api-access-gwxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:12:55.822522 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.822501 2559 scope.go:117] "RemoveContainer" containerID="d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da" Sep 17 19:12:55.829411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.829385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "057de55c-8402-4ca5-99d5-858100ed7520" (UID: "057de55c-8402-4ca5-99d5-858100ed7520"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:12:55.834213 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.834198 2559 scope.go:117] "RemoveContainer" containerID="b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08" Sep 17 19:12:55.840838 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.840824 2559 scope.go:117] "RemoveContainer" containerID="2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631" Sep 17 19:12:55.841081 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:55.841063 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631\": container with ID starting with 2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631 not found: ID does not exist" containerID="2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631" Sep 17 19:12:55.841124 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.841089 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631"} err="failed to get container status \"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631\": rpc error: code = NotFound desc = could not find container \"2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631\": container with ID starting with 2b80afc053caae98e85902bbbf110376ab9608cdeac3e901285d1332c9c13631 not found: ID does not exist" Sep 17 19:12:55.841124 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.841104 2559 scope.go:117] "RemoveContainer" containerID="d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da" Sep 17 19:12:55.841347 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:55.841331 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da\": container with ID starting with d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da not found: ID does not exist" containerID="d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da" Sep 17 19:12:55.841389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.841351 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da"} err="failed to get container status \"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da\": rpc error: code = NotFound desc = could not find container \"d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da\": container with ID starting with d958707ddd1ceb42698eb589f0a4804b976d8a36fe41db7c10dec230939ab8da not found: ID does not exist" Sep 17 19:12:55.841389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.841365 2559 scope.go:117] "RemoveContainer" containerID="b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08" Sep 17 19:12:55.841603 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:55.841585 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08\": container with ID starting with b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08 not found: ID does not exist" containerID="b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08" Sep 17 19:12:55.841648 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.841607 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08"} err="failed to get container status \"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08\": rpc error: code = NotFound desc = could not find container \"b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08\": container with ID starting with b0963ca26e0427b1e296858d46e52a43a65e1116c0059246b9144e0e37855b08 not found: ID does not exist" Sep 17 19:12:55.919713 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.919692 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxkf\" (UniqueName: \"kubernetes.io/projected/057de55c-8402-4ca5-99d5-858100ed7520-kube-api-access-gwxkf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:55.919789 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.919713 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:55.919789 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.919726 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057de55c-8402-4ca5-99d5-858100ed7520-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:55.919789 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.919735 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/057de55c-8402-4ca5-99d5-858100ed7520-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:55.919789 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:55.919744 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/057de55c-8402-4ca5-99d5-858100ed7520-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:12:56.130320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:56.130257 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:12:56.134621 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:56.134597 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-dfwx6"] Sep 17 19:12:57.992158 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:57.992120 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057de55c-8402-4ca5-99d5-858100ed7520" path="/var/lib/kubelet/pods/057de55c-8402-4ca5-99d5-858100ed7520/volumes" Sep 17 19:12:58.653112 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653077 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:12:58.653415 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653403 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653417 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653431 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="storage-initializer" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653437 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="storage-initializer" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653466 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653477 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" Sep 17 19:12:58.653491 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653491 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="storage-initializer" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653498 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="storage-initializer" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653503 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kube-rbac-proxy" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653509 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kube-rbac-proxy" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653517 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-agent" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653522 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-agent" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653529 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653534 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653588 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kserve-container" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653597 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-agent" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653604 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kube-rbac-proxy" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653609 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eeb9d99-567f-4a90-b5b0-adb7f68d0c9e" containerName="kserve-container" Sep 17 19:12:58.653722 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.653615 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="057de55c-8402-4ca5-99d5-858100ed7520" containerName="kube-rbac-proxy" Sep 17 19:12:58.656831 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.656814 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.660675 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.660650 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 19:12:58.660876 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.660850 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:12:58.660993 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.660932 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 19:12:58.661602 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.661586 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\"" Sep 17 19:12:58.661602 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.661599 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 19:12:58.661731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.661625 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 19:12:58.661731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.661628 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:12:58.714636 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.714613 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:12:58.740495 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.740470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.740583 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.740509 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.740621 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.740580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.740669 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.740648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.740707 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.740684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f887q\" (UniqueName: \"kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.841760 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.841735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.841846 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.841767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f887q\" (UniqueName: \"kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.841846 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.841785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.841925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.841893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.842010 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:58.841992 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found Sep 17 19:12:58.842051 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.842021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.842091 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:12:58.842064 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls podName:f88866b6-e675-4825-9e89-b465a80f8193 nodeName:}" failed. No retries permitted until 2026-09-17 19:12:59.342045958 +0000 UTC m=+2631.330662194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-h4gzf" (UID: "f88866b6-e675-4825-9e89-b465a80f8193") : secret "isvc-paddle-runtime-predictor-serving-cert" not found Sep 17 19:12:58.842236 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.842217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.842429 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.842412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.842541 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.842524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:58.855769 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:58.855749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f887q\" (UniqueName: \"kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:59.346700 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:59.346671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:59.349241 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:59.349220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-h4gzf\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:59.566799 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:59.566771 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:12:59.856588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:12:59.856424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:12:59.859201 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:12:59.859172 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88866b6_e675_4825_9e89_b465a80f8193.slice/crio-8737898c1f9b3bfc650826aea167ce2f6ded9510b3b31f135c487062b6cfff8d WatchSource:0}: Error finding container 8737898c1f9b3bfc650826aea167ce2f6ded9510b3b31f135c487062b6cfff8d: Status 404 returned error can't find the container with id 8737898c1f9b3bfc650826aea167ce2f6ded9510b3b31f135c487062b6cfff8d Sep 17 19:13:00.836323 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:00.836280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerStarted","Data":"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d"} Sep 17 19:13:00.836784 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:00.836354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerStarted","Data":"8737898c1f9b3bfc650826aea167ce2f6ded9510b3b31f135c487062b6cfff8d"} Sep 17 19:13:04.855623 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:04.855582 2559 generic.go:356] "Generic (PLEG): container finished" podID="f88866b6-e675-4825-9e89-b465a80f8193" containerID="dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d" exitCode=0 Sep 17 19:13:04.855969 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:04.855655 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerDied","Data":"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d"} Sep 17 19:13:05.861992 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.861961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerStarted","Data":"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf"} Sep 17 19:13:05.862361 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.862020 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerStarted","Data":"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117"} Sep 17 19:13:05.862361 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.862292 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:13:05.862488 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.862461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:13:05.863907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.863876 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:05.895125 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:05.895076 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podStartSLOduration=3.570898974 podStartE2EDuration="7.895060687s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.324161713s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:05.892843402 +0000 UTC m=+2637.881459651" watchObservedRunningTime="2026-09-17 19:13:05.895060687 +0000 UTC m=+2637.883676937" Sep 17 19:13:06.866866 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:06.866827 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:11.871792 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:11.871762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:13:11.872248 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:11.872222 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:14.329904 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.329869 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:13:14.332371 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.332354 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.337171 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.337154 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\"" Sep 17 19:13:14.337613 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.337587 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Sep 17 19:13:14.337613 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.337593 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\"" Sep 17 19:13:14.377394 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.377370 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:13:14.453032 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.453001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.453032 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.453030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.453197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.453050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwpx\" (UniqueName: \"kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.453197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.453157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.453197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.453185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.553857 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.553822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.553857 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.553852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.553880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.553919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.553951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpx\" (UniqueName: \"kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554276 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.554244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554644 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.554627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.554817 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.554794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.556509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.556480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.567357 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.567323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwpx\" (UniqueName: \"kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.642212 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.642194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:14.932876 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:14.932793 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:13:14.941350 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:13:14.941318 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6e7e57_68c7_46fe_9aed_a28bada61dba.slice/crio-feff2881b5fd5586fabd315ff29083a2f762a87de528439137c3a5ca5fe87ecd WatchSource:0}: Error finding container feff2881b5fd5586fabd315ff29083a2f762a87de528439137c3a5ca5fe87ecd: Status 404 returned error can't find the container with id feff2881b5fd5586fabd315ff29083a2f762a87de528439137c3a5ca5fe87ecd Sep 17 19:13:15.905552 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:15.905508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerStarted","Data":"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55"} Sep 17 19:13:15.905918 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:15.905588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerStarted","Data":"feff2881b5fd5586fabd315ff29083a2f762a87de528439137c3a5ca5fe87ecd"} Sep 17 19:13:16.910561 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:16.910526 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerID="db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55" exitCode=0 Sep 17 19:13:16.910929 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:16.910612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerDied","Data":"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55"} Sep 17 19:13:17.917411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:17.917376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerStarted","Data":"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0"} Sep 17 19:13:17.917901 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:17.917442 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerStarted","Data":"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84"} Sep 17 19:13:17.917901 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:17.917492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:17.959281 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:17.959230 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podStartSLOduration=2.947675678 podStartE2EDuration="3.95921632s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.011540642s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:17.959024739 +0000 UTC m=+2649.947640987" watchObservedRunningTime="2026-09-17 19:13:17.95921632 +0000 UTC m=+2649.947832948" Sep 17 19:13:18.922061 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:18.922032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:18.923328 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:18.923289 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:19.926534 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:19.926496 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:21.873110 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:21.873070 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:24.931341 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:24.931312 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:13:24.931799 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:24.931773 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:31.873061 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:31.873021 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:34.932396 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:34.932354 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:41.872197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:41.872149 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:13:44.932171 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:44.932120 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:51.873762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:51.873687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:13:54.932129 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:54.932087 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:13:59.814163 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:59.814129 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:13:59.814580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:59.814383 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" containerID="cri-o://b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117" gracePeriod=30 Sep 17 19:13:59.814580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:13:59.814407 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kube-rbac-proxy" containerID="cri-o://aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf" gracePeriod=30 Sep 17 19:14:00.100177 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:00.100098 2559 generic.go:356] "Generic (PLEG): container finished" podID="f88866b6-e675-4825-9e89-b465a80f8193" containerID="aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf" exitCode=2 Sep 17 19:14:00.100320 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:00.100167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerDied","Data":"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf"} Sep 17 19:14:01.868078 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:01.868039 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused" Sep 17 19:14:01.872922 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:01.872896 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused" Sep 17 19:14:02.349736 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.349711 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:14:02.518790 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.518718 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert\") pod \"f88866b6-e675-4825-9e89-b465a80f8193\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " Sep 17 19:14:02.518790 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.518759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location\") pod \"f88866b6-e675-4825-9e89-b465a80f8193\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " Sep 17 19:14:02.518790 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.518780 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") pod \"f88866b6-e675-4825-9e89-b465a80f8193\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " Sep 17 19:14:02.519006 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.518807 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config\") pod \"f88866b6-e675-4825-9e89-b465a80f8193\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " Sep 17 19:14:02.519006 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.518836 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f887q\" (UniqueName: \"kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q\") pod \"f88866b6-e675-4825-9e89-b465a80f8193\" (UID: \"f88866b6-e675-4825-9e89-b465a80f8193\") " Sep 17 19:14:02.519155 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.519125 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f88866b6-e675-4825-9e89-b465a80f8193" (UID: "f88866b6-e675-4825-9e89-b465a80f8193"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:14:02.519201 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.519159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f88866b6-e675-4825-9e89-b465a80f8193" (UID: "f88866b6-e675-4825-9e89-b465a80f8193"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:14:02.520879 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.520851 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f88866b6-e675-4825-9e89-b465a80f8193" (UID: "f88866b6-e675-4825-9e89-b465a80f8193"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:14:02.521001 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.520971 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q" (OuterVolumeSpecName: "kube-api-access-f887q") pod "f88866b6-e675-4825-9e89-b465a80f8193" (UID: "f88866b6-e675-4825-9e89-b465a80f8193"). InnerVolumeSpecName "kube-api-access-f887q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:14:02.527197 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.527172 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f88866b6-e675-4825-9e89-b465a80f8193" (UID: "f88866b6-e675-4825-9e89-b465a80f8193"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:14:02.619385 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.619361 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:02.619385 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.619382 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88866b6-e675-4825-9e89-b465a80f8193-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:02.619529 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.619391 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88866b6-e675-4825-9e89-b465a80f8193-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:02.619529 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.619399 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88866b6-e675-4825-9e89-b465a80f8193-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:02.619529 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:02.619408 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f887q\" (UniqueName: \"kubernetes.io/projected/f88866b6-e675-4825-9e89-b465a80f8193-kube-api-access-f887q\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:03.115030 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.114992 2559 generic.go:356] "Generic (PLEG): container finished" podID="f88866b6-e675-4825-9e89-b465a80f8193" containerID="b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117" exitCode=0 Sep 17 19:14:03.115417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.115071 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" Sep 17 19:14:03.115417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.115079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerDied","Data":"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117"} Sep 17 19:14:03.115417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.115142 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf" event={"ID":"f88866b6-e675-4825-9e89-b465a80f8193","Type":"ContainerDied","Data":"8737898c1f9b3bfc650826aea167ce2f6ded9510b3b31f135c487062b6cfff8d"} Sep 17 19:14:03.115417 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.115169 2559 scope.go:117] "RemoveContainer" containerID="aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf" Sep 17 19:14:03.124250 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.124233 2559 scope.go:117] "RemoveContainer" containerID="b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117" Sep 17 19:14:03.131723 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.131707 2559 scope.go:117] "RemoveContainer" containerID="dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d" Sep 17 19:14:03.139422 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.139397 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:14:03.139631 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.139615 2559 scope.go:117] "RemoveContainer" containerID="aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf" Sep 17 19:14:03.139906 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:03.139888 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf\": container with ID starting with aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf not found: ID does not exist" containerID="aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf" Sep 17 19:14:03.139954 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.139914 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf"} err="failed to get container status \"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf\": rpc error: code = NotFound desc = could not find container \"aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf\": container with ID starting with aae641a51b485d1787dc9ad673883aa0350aae6b62c40e64268325a2a9c7e1cf not found: ID does not exist" Sep 17 19:14:03.139954 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.139932 2559 scope.go:117] "RemoveContainer" containerID="b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117" Sep 17 19:14:03.140184 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:03.140159 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117\": container with ID starting with b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117 not found: ID does not exist" containerID="b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117" Sep 17 19:14:03.140270 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.140193 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117"} err="failed to get container status \"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117\": rpc error: code = NotFound desc = could not find container \"b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117\": container with ID starting with b64d916bdf31775a90667f471e7748c6992cf5c402cfc23075e7da93fb857117 not found: ID does not exist" Sep 17 19:14:03.140270 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.140232 2559 scope.go:117] "RemoveContainer" containerID="dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d" Sep 17 19:14:03.140504 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:03.140483 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d\": container with ID starting with dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d not found: ID does not exist" containerID="dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d" Sep 17 19:14:03.140611 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.140506 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d"} err="failed to get container status \"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d\": rpc error: code = NotFound desc = could not find container \"dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d\": container with ID starting with dd4f44cc62d939c0f3b0ad986ba08e9ac5b1f8c48a907c02c39ca300d4b5ce8d not found: ID does not exist" Sep 17 19:14:03.147144 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.147113 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-h4gzf"] Sep 17 19:14:03.997539 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:03.997505 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88866b6-e675-4825-9e89-b465a80f8193" path="/var/lib/kubelet/pods/f88866b6-e675-4825-9e89-b465a80f8193/volumes" Sep 17 19:14:04.932683 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:04.932645 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:14:05.135066 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135035 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:14:05.135360 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135349 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" Sep 17 19:14:05.135405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135362 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" Sep 17 19:14:05.135405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135376 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kube-rbac-proxy" Sep 17 19:14:05.135405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135382 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kube-rbac-proxy" Sep 17 19:14:05.135405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135396 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="storage-initializer" Sep 17 19:14:05.135405 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135402 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="storage-initializer" Sep 17 19:14:05.135586 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135466 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kserve-container" Sep 17 19:14:05.135586 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.135474 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88866b6-e675-4825-9e89-b465a80f8193" containerName="kube-rbac-proxy" Sep 17 19:14:05.140269 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.140253 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.143138 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.143117 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:14:05.143356 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.143341 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\"" Sep 17 19:14:05.143585 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.143565 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\"" Sep 17 19:14:05.150542 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.150521 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:14:05.237892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.237818 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7s2f\" (UniqueName: \"kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.237892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.237852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.237892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.237869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.237892 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.237889 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.238162 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.237941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.338907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.338866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.338907 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.338909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.338929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.338945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.339011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7s2f\" (UniqueName: \"kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339159 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:05.339036 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found Sep 17 19:14:05.339159 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:05.339107 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls podName:07d0c923-1f5e-4acc-b19a-d82f7097b1e4 nodeName:}" failed. No retries permitted until 2026-09-17 19:14:05.839085936 +0000 UTC m=+2697.827702174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found Sep 17 19:14:05.339416 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.339330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339661 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.339645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.339707 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.339649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.350810 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.350774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7s2f\" (UniqueName: \"kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.844930 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.844894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:05.847225 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:05.847203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:06.051715 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:06.051682 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:06.326399 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:06.326363 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:14:06.328309 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:14:06.328272 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d0c923_1f5e_4acc_b19a_d82f7097b1e4.slice/crio-b3ab4fbef45e1b9aa8e0555c0c3e55e9479ce51f128f2e2ae3aa94d5df38ccfd WatchSource:0}: Error finding container b3ab4fbef45e1b9aa8e0555c0c3e55e9479ce51f128f2e2ae3aa94d5df38ccfd: Status 404 returned error can't find the container with id b3ab4fbef45e1b9aa8e0555c0c3e55e9479ce51f128f2e2ae3aa94d5df38ccfd Sep 17 19:14:07.134694 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:07.134661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerStarted","Data":"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e"} Sep 17 19:14:07.135165 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:07.134718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerStarted","Data":"b3ab4fbef45e1b9aa8e0555c0c3e55e9479ce51f128f2e2ae3aa94d5df38ccfd"} Sep 17 19:14:10.121110 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:10.121080 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:14:10.123661 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:10.123640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:14:10.129630 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:10.129607 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:14:10.131845 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:10.131828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:14:11.152762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:11.152732 2559 generic.go:356] "Generic (PLEG): container finished" podID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerID="ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e" exitCode=0 Sep 17 19:14:11.153216 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:11.152781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerDied","Data":"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e"} Sep 17 19:14:12.160143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.160113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerStarted","Data":"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24"} Sep 17 19:14:12.160618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.160180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerStarted","Data":"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3"} Sep 17 19:14:12.160618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.160440 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:12.160618 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.160598 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:12.161871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.161844 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:12.188174 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:12.188118 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podStartSLOduration=2.9382738269999997 podStartE2EDuration="7.188102763s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.249828936s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:12.185806852 +0000 UTC m=+2704.174423101" watchObservedRunningTime="2026-09-17 19:14:12.188102763 +0000 UTC m=+2704.176719010" Sep 17 19:14:13.164337 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:13.164301 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:14.932651 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:14.932600 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:14:18.168905 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:18.168875 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:18.169344 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:18.169320 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:24.932125 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:24.932086 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:14:28.169866 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:28.169826 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:34.932468 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:34.932419 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:14:38.170256 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:38.170218 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:44.158092 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.158057 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:14:44.158483 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.158331 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" containerID="cri-o://767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84" gracePeriod=30 Sep 17 19:14:44.158728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.158700 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kube-rbac-proxy" containerID="cri-o://fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0" gracePeriod=30 Sep 17 19:14:44.295853 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.295828 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerID="fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0" exitCode=2 Sep 17 19:14:44.295970 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.295889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerDied","Data":"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0"} Sep 17 19:14:44.927126 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.927082 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused" Sep 17 19:14:44.932441 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:44.932414 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused" Sep 17 19:14:48.169427 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.169388 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:14:48.503409 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.503387 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:14:48.554057 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert\") pod \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " Sep 17 19:14:48.554142 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location\") pod \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " Sep 17 19:14:48.554142 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554095 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls\") pod \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " Sep 17 19:14:48.554142 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554122 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config\") pod \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " Sep 17 19:14:48.554314 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554157 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhwpx\" (UniqueName: \"kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx\") pod \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\" (UID: \"2e6e7e57-68c7-46fe-9aed-a28bada61dba\") " Sep 17 19:14:48.554401 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554372 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e6e7e57-68c7-46fe-9aed-a28bada61dba" (UID: "2e6e7e57-68c7-46fe-9aed-a28bada61dba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:14:48.554481 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554424 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e6e7e57-68c7-46fe-9aed-a28bada61dba" (UID: "2e6e7e57-68c7-46fe-9aed-a28bada61dba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:14:48.554540 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.554491 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e6e7e57-68c7-46fe-9aed-a28bada61dba" (UID: "2e6e7e57-68c7-46fe-9aed-a28bada61dba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:14:48.556201 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.556181 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e6e7e57-68c7-46fe-9aed-a28bada61dba" (UID: "2e6e7e57-68c7-46fe-9aed-a28bada61dba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:14:48.556271 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.556226 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx" (OuterVolumeSpecName: "kube-api-access-nhwpx") pod "2e6e7e57-68c7-46fe-9aed-a28bada61dba" (UID: "2e6e7e57-68c7-46fe-9aed-a28bada61dba"). InnerVolumeSpecName "kube-api-access-nhwpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:14:48.655742 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.655717 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:48.655742 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.655740 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwpx\" (UniqueName: \"kubernetes.io/projected/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kube-api-access-nhwpx\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:48.655866 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.655750 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e7e57-68c7-46fe-9aed-a28bada61dba-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:48.655866 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.655758 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6e7e57-68c7-46fe-9aed-a28bada61dba-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:48.655866 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:48.655767 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6e7e57-68c7-46fe-9aed-a28bada61dba-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:14:49.318366 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.318330 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerID="767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84" exitCode=0 Sep 17 19:14:49.318765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.318388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerDied","Data":"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84"} Sep 17 19:14:49.318765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.318441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" event={"ID":"2e6e7e57-68c7-46fe-9aed-a28bada61dba","Type":"ContainerDied","Data":"feff2881b5fd5586fabd315ff29083a2f762a87de528439137c3a5ca5fe87ecd"} Sep 17 19:14:49.318765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.318393 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw" Sep 17 19:14:49.318765 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.318494 2559 scope.go:117] "RemoveContainer" containerID="fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0" Sep 17 19:14:49.331609 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.331582 2559 scope.go:117] "RemoveContainer" containerID="767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84" Sep 17 19:14:49.339377 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.339362 2559 scope.go:117] "RemoveContainer" containerID="db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55" Sep 17 19:14:49.346737 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.346716 2559 scope.go:117] "RemoveContainer" containerID="fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0" Sep 17 19:14:49.347046 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:49.347023 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0\": container with ID starting with fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0 not found: ID does not exist" containerID="fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0" Sep 17 19:14:49.347109 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.347058 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0"} err="failed to get container status \"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0\": rpc error: code = NotFound desc = could not find container \"fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0\": container with ID starting with fd9e607874d55507599a702d1995c993466b3c8e46480c395a068a28d01090e0 not found: ID does not exist" Sep 17 19:14:49.347109 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.347080 2559 scope.go:117] "RemoveContainer" containerID="767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84" Sep 17 19:14:49.347384 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:49.347362 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84\": container with ID starting with 767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84 not found: ID does not exist" containerID="767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84" Sep 17 19:14:49.347478 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.347392 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84"} err="failed to get container status \"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84\": rpc error: code = NotFound desc = could not find container \"767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84\": container with ID starting with 767eb19fbebb79ca6d31907922799899d9b316ee3c09bd5246427beee98cda84 not found: ID does not exist" Sep 17 19:14:49.347478 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.347413 2559 scope.go:117] "RemoveContainer" containerID="db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55" Sep 17 19:14:49.347712 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:49.347691 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55\": container with ID starting with db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55 not found: ID does not exist" containerID="db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55" Sep 17 19:14:49.347756 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.347720 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55"} err="failed to get container status \"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55\": rpc error: code = NotFound desc = could not find container \"db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55\": container with ID starting with db24a6d217bcff919d7e675f610557e4a9f991b96c8f6a29f55a7de5b2046c55 not found: ID does not exist" Sep 17 19:14:49.348239 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.348223 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:14:49.352209 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.352187 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6b698696c6-8h9kw"] Sep 17 19:14:49.992673 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:49.992641 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" path="/var/lib/kubelet/pods/2e6e7e57-68c7-46fe-9aed-a28bada61dba/volumes" Sep 17 19:14:56.417103 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417064 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417359 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kube-rbac-proxy" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417370 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kube-rbac-proxy" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417382 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="storage-initializer" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417387 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="storage-initializer" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417403 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417409 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417494 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kserve-container" Sep 17 19:14:56.417728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.417504 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6e7e57-68c7-46fe-9aed-a28bada61dba" containerName="kube-rbac-proxy" Sep 17 19:14:56.422161 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.422143 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.425302 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.425280 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Sep 17 19:14:56.427538 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.427516 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\"" Sep 17 19:14:56.427640 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.427625 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\"" Sep 17 19:14:56.432245 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.432222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:14:56.515777 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.515749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.515912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.515794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.515912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.515827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbxl\" (UniqueName: \"kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.515912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.515858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.515912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.515893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.616991 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.616953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbxl\" (UniqueName: \"kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.616991 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.616997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617172 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617172 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617172 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617391 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:56.617180 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found Sep 17 19:14:56.617391 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:14:56.617240 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls podName:29757c16-6d89-49b0-9e00-f673bcf4b3dc nodeName:}" failed. No retries permitted until 2026-09-17 19:14:57.117220145 +0000 UTC m=+2749.105836389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found Sep 17 19:14:56.617538 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617823 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.617901 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.617798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:56.628330 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:56.628306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbxl\" (UniqueName: \"kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:57.121498 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:57.121443 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:57.123725 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:57.123694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:57.332536 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:57.332500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:14:57.604924 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:57.604901 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:14:57.606836 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:14:57.606804 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29757c16_6d89_49b0_9e00_f673bcf4b3dc.slice/crio-7f7d25e8c17b86fa86aae7b142a6ba8b909cc6c477a259b5fbe7073357fbc97e WatchSource:0}: Error finding container 7f7d25e8c17b86fa86aae7b142a6ba8b909cc6c477a259b5fbe7073357fbc97e: Status 404 returned error can't find the container with id 7f7d25e8c17b86fa86aae7b142a6ba8b909cc6c477a259b5fbe7073357fbc97e Sep 17 19:14:58.170813 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:58.170781 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:14:58.360515 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:58.360479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerStarted","Data":"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6"} Sep 17 19:14:58.360695 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:58.360538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerStarted","Data":"7f7d25e8c17b86fa86aae7b142a6ba8b909cc6c477a259b5fbe7073357fbc97e"} Sep 17 19:14:59.365899 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:59.365863 2559 generic.go:356] "Generic (PLEG): container finished" podID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerID="67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6" exitCode=0 Sep 17 19:14:59.366281 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:14:59.365944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerDied","Data":"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6"} Sep 17 19:15:00.372520 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.372483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerStarted","Data":"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d"} Sep 17 19:15:00.372945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.372570 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:15:00.372945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.372594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerStarted","Data":"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24"} Sep 17 19:15:00.372945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.372613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:15:00.374095 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.374067 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:00.393920 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:00.393877 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podStartSLOduration=3.368617797 podStartE2EDuration="4.393863764s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.025245967s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:00.392644202 +0000 UTC m=+2752.381260451" watchObservedRunningTime="2026-09-17 19:15:00.393863764 +0000 UTC m=+2752.382480013" Sep 17 19:15:01.377414 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:01.377369 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:06.382881 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:06.382850 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:15:06.383347 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:06.383321 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:06.755752 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:06.755716 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:15:06.755980 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:06.755952 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" containerID="cri-o://d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3" gracePeriod=30 Sep 17 19:15:06.756059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:06.755981 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kube-rbac-proxy" containerID="cri-o://e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24" gracePeriod=30 Sep 17 19:15:07.404442 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:07.404407 2559 generic.go:356] "Generic (PLEG): container finished" podID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerID="e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24" exitCode=2 Sep 17 19:15:07.404811 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:07.404487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerDied","Data":"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24"} Sep 17 19:15:08.165643 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:08.165602 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused" Sep 17 19:15:08.170050 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:08.170024 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused" Sep 17 19:15:09.697919 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.697896 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:15:09.821460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821371 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config\") pod \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " Sep 17 19:15:09.821460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") pod \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " Sep 17 19:15:09.821460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821457 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert\") pod \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " Sep 17 19:15:09.821747 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821483 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location\") pod \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " Sep 17 19:15:09.821747 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821675 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7s2f\" (UniqueName: \"kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f\") pod \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\" (UID: \"07d0c923-1f5e-4acc-b19a-d82f7097b1e4\") " Sep 17 19:15:09.821862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07d0c923-1f5e-4acc-b19a-d82f7097b1e4" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:15:09.821862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821825 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07d0c923-1f5e-4acc-b19a-d82f7097b1e4" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:15:09.821988 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821977 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:15:09.822026 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.821996 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:15:09.823605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.823580 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07d0c923-1f5e-4acc-b19a-d82f7097b1e4" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:15:09.823773 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.823755 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f" (OuterVolumeSpecName: "kube-api-access-b7s2f") pod "07d0c923-1f5e-4acc-b19a-d82f7097b1e4" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4"). InnerVolumeSpecName "kube-api-access-b7s2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:15:09.831164 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.831142 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07d0c923-1f5e-4acc-b19a-d82f7097b1e4" (UID: "07d0c923-1f5e-4acc-b19a-d82f7097b1e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:15:09.922587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.922565 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:15:09.922587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.922585 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7s2f\" (UniqueName: \"kubernetes.io/projected/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-kube-api-access-b7s2f\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:15:09.922731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:09.922594 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d0c923-1f5e-4acc-b19a-d82f7097b1e4-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:15:10.419403 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.419366 2559 generic.go:356] "Generic (PLEG): container finished" podID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerID="d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3" exitCode=0 Sep 17 19:15:10.419567 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.419436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" Sep 17 19:15:10.419567 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.419469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerDied","Data":"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3"} Sep 17 19:15:10.419567 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.419536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc" event={"ID":"07d0c923-1f5e-4acc-b19a-d82f7097b1e4","Type":"ContainerDied","Data":"b3ab4fbef45e1b9aa8e0555c0c3e55e9479ce51f128f2e2ae3aa94d5df38ccfd"} Sep 17 19:15:10.419567 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.419563 2559 scope.go:117] "RemoveContainer" containerID="e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24" Sep 17 19:15:10.427768 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.427740 2559 scope.go:117] "RemoveContainer" containerID="d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3" Sep 17 19:15:10.434891 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.434875 2559 scope.go:117] "RemoveContainer" containerID="ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e" Sep 17 19:15:10.441575 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.440728 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:15:10.445187 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.445163 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-cgvvc"] Sep 17 19:15:10.445743 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.445728 2559 scope.go:117] "RemoveContainer" containerID="e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24" Sep 17 19:15:10.446022 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:15:10.446005 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24\": container with ID starting with e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24 not found: ID does not exist" containerID="e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24" Sep 17 19:15:10.446093 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.446033 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24"} err="failed to get container status \"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24\": rpc error: code = NotFound desc = could not find container \"e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24\": container with ID starting with e59cb8463e375ba5d158ce6269fcb6d0c1ad273779b64d9258b70e1a82704f24 not found: ID does not exist" Sep 17 19:15:10.446093 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.446058 2559 scope.go:117] "RemoveContainer" containerID="d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3" Sep 17 19:15:10.446322 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:15:10.446304 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3\": container with ID starting with d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3 not found: ID does not exist" containerID="d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3" Sep 17 19:15:10.446362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.446327 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3"} err="failed to get container status \"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3\": rpc error: code = NotFound desc = could not find container \"d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3\": container with ID starting with d4d341d4beb5465bf5b6864e680cd22560ed42898a9c42b3b25e47fe6fb673f3 not found: ID does not exist" Sep 17 19:15:10.446362 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.446343 2559 scope.go:117] "RemoveContainer" containerID="ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e" Sep 17 19:15:10.446578 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:15:10.446557 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e\": container with ID starting with ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e not found: ID does not exist" containerID="ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e" Sep 17 19:15:10.446638 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:10.446585 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e"} err="failed to get container status \"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e\": rpc error: code = NotFound desc = could not find container \"ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e\": container with ID starting with ad06ffc29250dc34dee78989862e0d22cfbe8dbab9efbc39a96ac6f67c98634e not found: ID does not exist" Sep 17 19:15:11.874693 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874660 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874953 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874964 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874980 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kube-rbac-proxy" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874985 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kube-rbac-proxy" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.874996 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="storage-initializer" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.875002 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="storage-initializer" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.875052 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kserve-container" Sep 17 19:15:11.876938 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.875061 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" containerName="kube-rbac-proxy" Sep 17 19:15:11.877888 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.877873 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:11.881106 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.881086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:15:11.881587 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.881569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\"" Sep 17 19:15:11.881641 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.881591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\"" Sep 17 19:15:11.896674 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.896652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:15:11.993212 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:11.993184 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d0c923-1f5e-4acc-b19a-d82f7097b1e4" path="/var/lib/kubelet/pods/07d0c923-1f5e-4acc-b19a-d82f7097b1e4/volumes" Sep 17 19:15:12.042395 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.042371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.042565 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.042400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.042565 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.042423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.042565 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.042475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.042565 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.042530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dl57\" (UniqueName: \"kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.143698 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.143674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.143871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.143705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.143871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.143730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.143871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.143765 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.143871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.143820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dl57\" (UniqueName: \"kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.144321 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.144294 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.144531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.144507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.144531 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.144524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.146199 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.146179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.159010 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.158974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dl57\" (UniqueName: \"kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57\") pod \"isvc-pmml-predictor-8674795f59-f4twl\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.188066 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.188041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:12.503982 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:12.503910 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:15:12.507243 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:15:12.507216 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6320665f_64ef_47be_9880_7dacf6e45c62.slice/crio-c2e01d8bc2836088972e18de36aca4ad64268776fb14d41464ba199d73e61943 WatchSource:0}: Error finding container c2e01d8bc2836088972e18de36aca4ad64268776fb14d41464ba199d73e61943: Status 404 returned error can't find the container with id c2e01d8bc2836088972e18de36aca4ad64268776fb14d41464ba199d73e61943 Sep 17 19:15:13.436259 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:13.436214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerStarted","Data":"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131"} Sep 17 19:15:13.436671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:13.436286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerStarted","Data":"c2e01d8bc2836088972e18de36aca4ad64268776fb14d41464ba199d73e61943"} Sep 17 19:15:16.384146 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:16.384116 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:16.450799 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:16.450771 2559 generic.go:356] "Generic (PLEG): container finished" podID="6320665f-64ef-47be-9880-7dacf6e45c62" containerID="02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131" exitCode=0 Sep 17 19:15:16.450915 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:16.450824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerDied","Data":"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131"} Sep 17 19:15:16.452101 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:16.452076 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 19:15:24.495143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:24.495109 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerStarted","Data":"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1"} Sep 17 19:15:24.495535 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:24.495175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerStarted","Data":"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60"} Sep 17 19:15:24.495535 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:24.495372 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:24.519229 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:24.519185 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podStartSLOduration=2.282384593 podStartE2EDuration="13.519171066s" totalImagesPullingTime="7.502583715s" totalInitContainerRuntime="3.734202758s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:24.516624573 +0000 UTC m=+2776.505240821" watchObservedRunningTime="2026-09-17 19:15:24.519171066 +0000 UTC m=+2776.507787314" Sep 17 19:15:25.499167 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:25.499136 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:25.500552 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:25.500525 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:15:26.383359 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:26.383325 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:26.503883 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:26.503845 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:15:31.509519 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:31.509491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:15:31.510044 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:31.509980 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:15:36.384201 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:36.384157 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:41.510212 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:41.510170 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:15:46.383379 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:46.383339 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:15:51.510148 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:51.510109 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:15:56.383677 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:15:56.383640 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:16:01.509933 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:01.509894 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:06.383965 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:06.383926 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused" Sep 17 19:16:11.510721 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:11.510686 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:16.384610 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:16.384582 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:16:21.509988 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:21.509950 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:26.432756 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:26.432715 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:16:26.433180 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:26.433026 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" containerID="cri-o://e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24" gracePeriod=30 Sep 17 19:16:26.433180 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:26.433120 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kube-rbac-proxy" containerID="cri-o://99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d" gracePeriod=30 Sep 17 19:16:26.765628 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:26.765546 2559 generic.go:356] "Generic (PLEG): container finished" podID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerID="99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d" exitCode=2 Sep 17 19:16:26.765788 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:26.765620 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerDied","Data":"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d"} Sep 17 19:16:31.070571 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.070545 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:16:31.190816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.190780 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config\") pod \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " Sep 17 19:16:31.191003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.190828 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location\") pod \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " Sep 17 19:16:31.191003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.190869 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert\") pod \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " Sep 17 19:16:31.191003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.190914 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") pod \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " Sep 17 19:16:31.191003 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.190939 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbxl\" (UniqueName: \"kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl\") pod \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\" (UID: \"29757c16-6d89-49b0-9e00-f673bcf4b3dc\") " Sep 17 19:16:31.191217 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.191182 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29757c16-6d89-49b0-9e00-f673bcf4b3dc" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:16:31.191277 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.191213 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29757c16-6d89-49b0-9e00-f673bcf4b3dc" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:16:31.191337 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.191276 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29757c16-6d89-49b0-9e00-f673bcf4b3dc" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:16:31.193195 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.193172 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29757c16-6d89-49b0-9e00-f673bcf4b3dc" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:16:31.193195 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.193194 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl" (OuterVolumeSpecName: "kube-api-access-sgbxl") pod "29757c16-6d89-49b0-9e00-f673bcf4b3dc" (UID: "29757c16-6d89-49b0-9e00-f673bcf4b3dc"). InnerVolumeSpecName "kube-api-access-sgbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:16:31.292325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.292281 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:16:31.292325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.292320 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29757c16-6d89-49b0-9e00-f673bcf4b3dc-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:16:31.292325 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.292330 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbxl\" (UniqueName: \"kubernetes.io/projected/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-api-access-sgbxl\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:16:31.292563 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.292340 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:16:31.292563 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.292349 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29757c16-6d89-49b0-9e00-f673bcf4b3dc-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:16:31.510288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.510193 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:31.788941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.788852 2559 generic.go:356] "Generic (PLEG): container finished" podID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerID="e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24" exitCode=0 Sep 17 19:16:31.788941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.788905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerDied","Data":"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24"} Sep 17 19:16:31.788941 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.788915 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" Sep 17 19:16:31.789169 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.788964 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9" event={"ID":"29757c16-6d89-49b0-9e00-f673bcf4b3dc","Type":"ContainerDied","Data":"7f7d25e8c17b86fa86aae7b142a6ba8b909cc6c477a259b5fbe7073357fbc97e"} Sep 17 19:16:31.789169 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.789000 2559 scope.go:117] "RemoveContainer" containerID="99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d" Sep 17 19:16:31.798437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.798418 2559 scope.go:117] "RemoveContainer" containerID="e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24" Sep 17 19:16:31.805756 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.805737 2559 scope.go:117] "RemoveContainer" containerID="67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6" Sep 17 19:16:31.812494 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.812470 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:16:31.813385 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.813355 2559 scope.go:117] "RemoveContainer" containerID="99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d" Sep 17 19:16:31.813676 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:16:31.813658 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d\": container with ID starting with 99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d not found: ID does not exist" containerID="99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d" Sep 17 19:16:31.813728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.813684 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d"} err="failed to get container status \"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d\": rpc error: code = NotFound desc = could not find container \"99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d\": container with ID starting with 99430d49974e78bdbc9b4b67c28e002cb54287d7d6c9dd2ca0b245e6af49559d not found: ID does not exist" Sep 17 19:16:31.813728 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.813703 2559 scope.go:117] "RemoveContainer" containerID="e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24" Sep 17 19:16:31.813950 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:16:31.813933 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24\": container with ID starting with e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24 not found: ID does not exist" containerID="e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24" Sep 17 19:16:31.813988 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.813965 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24"} err="failed to get container status \"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24\": rpc error: code = NotFound desc = could not find container \"e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24\": container with ID starting with e8084388d5e59a241ba13472bc558e9eec1d02c3c1029ee0831d125c068a1f24 not found: ID does not exist" Sep 17 19:16:31.813988 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.813982 2559 scope.go:117] "RemoveContainer" containerID="67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6" Sep 17 19:16:31.814204 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:16:31.814188 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6\": container with ID starting with 67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6 not found: ID does not exist" containerID="67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6" Sep 17 19:16:31.814263 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.814208 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6"} err="failed to get container status \"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6\": rpc error: code = NotFound desc = could not find container \"67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6\": container with ID starting with 67c78bb52cb7bdaf1a90d8107dbb6d7da382ce90a2142ff083a864e4a2f21bd6 not found: ID does not exist" Sep 17 19:16:31.817056 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.817031 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-58c77b7cbd-kfmx9"] Sep 17 19:16:31.993007 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:31.992974 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" path="/var/lib/kubelet/pods/29757c16-6d89-49b0-9e00-f673bcf4b3dc/volumes" Sep 17 19:16:38.988314 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:38.988268 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:48.988632 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:48.988590 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused" Sep 17 19:16:58.988691 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:16:58.988613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:17:03.793539 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:03.793506 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:17:03.793919 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:03.793736 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" containerID="cri-o://2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60" gracePeriod=30 Sep 17 19:17:03.793919 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:03.793842 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kube-rbac-proxy" containerID="cri-o://561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1" gracePeriod=30 Sep 17 19:17:03.924973 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:03.924944 2559 generic.go:356] "Generic (PLEG): container finished" podID="6320665f-64ef-47be-9880-7dacf6e45c62" containerID="561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1" exitCode=2 Sep 17 19:17:03.925112 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:03.925010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerDied","Data":"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1"} Sep 17 19:17:06.504656 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:06.504607 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused" Sep 17 19:17:08.139324 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.139300 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:17:08.185997 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.185965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location\") pod \"6320665f-64ef-47be-9880-7dacf6e45c62\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " Sep 17 19:17:08.186159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186014 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dl57\" (UniqueName: \"kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57\") pod \"6320665f-64ef-47be-9880-7dacf6e45c62\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " Sep 17 19:17:08.186159 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186068 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls\") pod \"6320665f-64ef-47be-9880-7dacf6e45c62\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " Sep 17 19:17:08.186280 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186217 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6320665f-64ef-47be-9880-7dacf6e45c62" (UID: "6320665f-64ef-47be-9880-7dacf6e45c62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:17:08.186280 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186265 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert\") pod \"6320665f-64ef-47be-9880-7dacf6e45c62\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " Sep 17 19:17:08.186389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config\") pod \"6320665f-64ef-47be-9880-7dacf6e45c62\" (UID: \"6320665f-64ef-47be-9880-7dacf6e45c62\") " Sep 17 19:17:08.186582 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186553 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6320665f-64ef-47be-9880-7dacf6e45c62-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:17:08.186737 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186707 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6320665f-64ef-47be-9880-7dacf6e45c62" (UID: "6320665f-64ef-47be-9880-7dacf6e45c62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:17:08.186797 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.186735 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6320665f-64ef-47be-9880-7dacf6e45c62" (UID: "6320665f-64ef-47be-9880-7dacf6e45c62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:17:08.188190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.188168 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57" (OuterVolumeSpecName: "kube-api-access-6dl57") pod "6320665f-64ef-47be-9880-7dacf6e45c62" (UID: "6320665f-64ef-47be-9880-7dacf6e45c62"). InnerVolumeSpecName "kube-api-access-6dl57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:17:08.188317 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.188294 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6320665f-64ef-47be-9880-7dacf6e45c62" (UID: "6320665f-64ef-47be-9880-7dacf6e45c62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:17:08.287186 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.287106 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dl57\" (UniqueName: \"kubernetes.io/projected/6320665f-64ef-47be-9880-7dacf6e45c62-kube-api-access-6dl57\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:17:08.287186 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.287134 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6320665f-64ef-47be-9880-7dacf6e45c62-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:17:08.287186 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.287144 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:17:08.287186 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.287154 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6320665f-64ef-47be-9880-7dacf6e45c62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:17:08.948463 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.948415 2559 generic.go:356] "Generic (PLEG): container finished" podID="6320665f-64ef-47be-9880-7dacf6e45c62" containerID="2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60" exitCode=0 Sep 17 19:17:08.948669 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.948483 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" Sep 17 19:17:08.948669 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.948505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerDied","Data":"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60"} Sep 17 19:17:08.948669 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.948583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl" event={"ID":"6320665f-64ef-47be-9880-7dacf6e45c62","Type":"ContainerDied","Data":"c2e01d8bc2836088972e18de36aca4ad64268776fb14d41464ba199d73e61943"} Sep 17 19:17:08.948669 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.948621 2559 scope.go:117] "RemoveContainer" containerID="561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1" Sep 17 19:17:08.957902 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.957886 2559 scope.go:117] "RemoveContainer" containerID="2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60" Sep 17 19:17:08.965293 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.965275 2559 scope.go:117] "RemoveContainer" containerID="02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131" Sep 17 19:17:08.971862 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.971839 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:17:08.972657 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.972640 2559 scope.go:117] "RemoveContainer" containerID="561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1" Sep 17 19:17:08.972925 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:17:08.972898 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1\": container with ID starting with 561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1 not found: ID does not exist" containerID="561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1" Sep 17 19:17:08.972993 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.972924 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1"} err="failed to get container status \"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1\": rpc error: code = NotFound desc = could not find container \"561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1\": container with ID starting with 561c53583c377e2a6e1723c8ab123c951a22bbfe7ea35be62fb84a1380bc58f1 not found: ID does not exist" Sep 17 19:17:08.972993 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.972942 2559 scope.go:117] "RemoveContainer" containerID="2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60" Sep 17 19:17:08.973186 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:17:08.973167 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60\": container with ID starting with 2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60 not found: ID does not exist" containerID="2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60" Sep 17 19:17:08.973231 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.973192 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60"} err="failed to get container status \"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60\": rpc error: code = NotFound desc = could not find container \"2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60\": container with ID starting with 2b66cb69593c8c3938aba853abc02f186fd5db43234af6ab58e7061023beca60 not found: ID does not exist" Sep 17 19:17:08.973231 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.973208 2559 scope.go:117] "RemoveContainer" containerID="02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131" Sep 17 19:17:08.973465 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:17:08.973436 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131\": container with ID starting with 02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131 not found: ID does not exist" containerID="02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131" Sep 17 19:17:08.973512 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.973471 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131"} err="failed to get container status \"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131\": rpc error: code = NotFound desc = could not find container \"02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131\": container with ID starting with 02b04d781bd7009f3b6e2f00f689d3c410cede25f361455ee96f30d646ec9131 not found: ID does not exist" Sep 17 19:17:08.978501 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.978480 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-f4twl"] Sep 17 19:17:08.985460 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985433 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:17:08.985720 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985709 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985722 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985732 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kube-rbac-proxy" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985737 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kube-rbac-proxy" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985746 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="storage-initializer" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985752 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="storage-initializer" Sep 17 19:17:08.985762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985763 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kube-rbac-proxy" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kube-rbac-proxy" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985777 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="storage-initializer" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985782 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="storage-initializer" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985793 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985798 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985842 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kube-rbac-proxy" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985849 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kserve-container" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985855 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" containerName="kube-rbac-proxy" Sep 17 19:17:08.985945 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.985862 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="29757c16-6d89-49b0-9e00-f673bcf4b3dc" containerName="kserve-container" Sep 17 19:17:08.990464 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.990431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:08.994571 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.994552 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 19:17:08.994718 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.994610 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\"" Sep 17 19:17:08.994718 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.994612 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 19:17:08.994895 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.994797 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:17:08.995140 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.995125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\"" Sep 17 19:17:08.995238 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.995129 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 19:17:08.995339 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:08.995325 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 19:17:09.003158 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.003134 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:17:09.093925 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.093894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.094366 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.094326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.094540 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.094435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.094742 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.094698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.094867 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.094755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195324 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195324 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195796 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195796 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195796 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.195917 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.195823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.196182 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.196160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.196250 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.196160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.197906 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.197887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.205707 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.205657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzzht\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.300768 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.300737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:09.603607 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.603520 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:17:09.607132 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:17:09.607100 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5e9fa1_b3a5_4b78_8c30_74f1f3d5c52f.slice/crio-7cdead0997f55a49b606f4d936d7d45ee3c87bfea951d77541a69371e81c54ca WatchSource:0}: Error finding container 7cdead0997f55a49b606f4d936d7d45ee3c87bfea951d77541a69371e81c54ca: Status 404 returned error can't find the container with id 7cdead0997f55a49b606f4d936d7d45ee3c87bfea951d77541a69371e81c54ca Sep 17 19:17:09.961236 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.961152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerStarted","Data":"2507bf0043a2f8cbe51da8e434b575ca7e205007d2a6de6d4a1eac7960e24af1"} Sep 17 19:17:09.961412 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.961376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerStarted","Data":"7cdead0997f55a49b606f4d936d7d45ee3c87bfea951d77541a69371e81c54ca"} Sep 17 19:17:09.992865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:09.992836 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6320665f-64ef-47be-9880-7dacf6e45c62" path="/var/lib/kubelet/pods/6320665f-64ef-47be-9880-7dacf6e45c62/volumes" Sep 17 19:17:13.982182 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:13.982148 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerID="2507bf0043a2f8cbe51da8e434b575ca7e205007d2a6de6d4a1eac7960e24af1" exitCode=0 Sep 17 19:17:13.982630 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:13.982190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerDied","Data":"2507bf0043a2f8cbe51da8e434b575ca7e205007d2a6de6d4a1eac7960e24af1"} Sep 17 19:17:14.988286 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:14.988251 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerStarted","Data":"334b6e9007f9dac9e18e711b76dfb71dd01420723b80a9ce40fef4dd9517a993"} Sep 17 19:17:14.988689 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:14.988311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerStarted","Data":"aef486794cd6ea196283051d34c7ab33ebbc49b6227e9e7be3d729a62f69b023"} Sep 17 19:17:14.988689 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:14.988559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:15.011869 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:15.011820 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podStartSLOduration=3.328772106 podStartE2EDuration="7.011805805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683033699s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:15.010149904 +0000 UTC m=+2886.998766163" watchObservedRunningTime="2026-09-17 19:17:15.011805805 +0000 UTC m=+2887.000422054" Sep 17 19:17:15.992512 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:15.992479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:15.993889 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:15.993860 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:17:16.996190 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:16.996140 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:17:22.001383 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:22.001357 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:17:22.001836 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:22.001813 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:17:32.002424 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:32.002387 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:17:42.002048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:42.002010 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:17:52.002684 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:17:52.002638 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:02.002108 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:02.002065 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:12.002530 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:12.002489 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:22.002143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:22.002063 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:32.002411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:32.002370 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:33.988059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:33.988019 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:43.991779 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:43.991751 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:18:50.478940 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:50.478904 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:18:50.479389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:50.479190 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" containerID="cri-o://aef486794cd6ea196283051d34c7ab33ebbc49b6227e9e7be3d729a62f69b023" gracePeriod=30 Sep 17 19:18:50.479389 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:50.479239 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kube-rbac-proxy" containerID="cri-o://334b6e9007f9dac9e18e711b76dfb71dd01420723b80a9ce40fef4dd9517a993" gracePeriod=30 Sep 17 19:18:51.386653 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:51.386620 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerID="334b6e9007f9dac9e18e711b76dfb71dd01420723b80a9ce40fef4dd9517a993" exitCode=2 Sep 17 19:18:51.386824 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:51.386690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerDied","Data":"334b6e9007f9dac9e18e711b76dfb71dd01420723b80a9ce40fef4dd9517a993"} Sep 17 19:18:51.996803 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:51.996766 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused" Sep 17 19:18:53.988699 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:53.988656 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused" Sep 17 19:18:54.404437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.404410 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerID="aef486794cd6ea196283051d34c7ab33ebbc49b6227e9e7be3d729a62f69b023" exitCode=0 Sep 17 19:18:54.404590 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.404479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerDied","Data":"aef486794cd6ea196283051d34c7ab33ebbc49b6227e9e7be3d729a62f69b023"} Sep 17 19:18:54.517964 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.517937 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:18:54.577281 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577252 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52\") pod \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " Sep 17 19:18:54.577437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577296 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls\") pod \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " Sep 17 19:18:54.577437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config\") pod \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " Sep 17 19:18:54.577437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577339 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location\") pod \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " Sep 17 19:18:54.577437 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577375 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert\") pod \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\" (UID: \"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f\") " Sep 17 19:18:54.577731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577692 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" (UID: "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:18:54.577808 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577739 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" (UID: "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:18:54.577865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.577826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" (UID: "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:18:54.579408 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.579389 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52" (OuterVolumeSpecName: "kube-api-access-6xf52") pod "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" (UID: "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f"). InnerVolumeSpecName "kube-api-access-6xf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:18:54.579476 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.579420 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" (UID: "9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:18:54.678798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.678760 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-api-access-6xf52\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:18:54.678798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.678797 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:18:54.678798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.678807 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:18:54.679011 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.678818 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:18:54.679011 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:54.678827 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:18:55.411018 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.410993 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" Sep 17 19:18:55.411440 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.410988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht" event={"ID":"9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f","Type":"ContainerDied","Data":"7cdead0997f55a49b606f4d936d7d45ee3c87bfea951d77541a69371e81c54ca"} Sep 17 19:18:55.411440 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.411129 2559 scope.go:117] "RemoveContainer" containerID="334b6e9007f9dac9e18e711b76dfb71dd01420723b80a9ce40fef4dd9517a993" Sep 17 19:18:55.426827 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.426807 2559 scope.go:117] "RemoveContainer" containerID="aef486794cd6ea196283051d34c7ab33ebbc49b6227e9e7be3d729a62f69b023" Sep 17 19:18:55.434440 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.434423 2559 scope.go:117] "RemoveContainer" containerID="2507bf0043a2f8cbe51da8e434b575ca7e205007d2a6de6d4a1eac7960e24af1" Sep 17 19:18:55.449893 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.449870 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:18:55.454574 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.454553 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzzht"] Sep 17 19:18:55.837292 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837213 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:18:55.837556 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837543 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kube-rbac-proxy" Sep 17 19:18:55.837605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837558 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kube-rbac-proxy" Sep 17 19:18:55.837605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837566 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="storage-initializer" Sep 17 19:18:55.837605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837571 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="storage-initializer" Sep 17 19:18:55.837605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837584 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" Sep 17 19:18:55.837605 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837590 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" Sep 17 19:18:55.837762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837644 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kserve-container" Sep 17 19:18:55.837762 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.837653 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" containerName="kube-rbac-proxy" Sep 17 19:18:55.842176 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.842160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.847561 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.847539 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jrhtg\"" Sep 17 19:18:55.847807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.847786 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 19:18:55.847926 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.847861 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\"" Sep 17 19:18:55.848350 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.848322 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\"" Sep 17 19:18:55.848775 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.848757 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 19:18:55.848847 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.848828 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 19:18:55.853224 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.853206 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 19:18:55.860346 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.860323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:18:55.887568 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.887536 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.887690 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.887572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.887690 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.887648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.887816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.887716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg77z\" (UniqueName: \"kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.887816 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.887734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988218 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988218 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988485 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988485 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988274 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg77z\" (UniqueName: \"kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988485 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.988745 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.988726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.989051 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.989030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.989097 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.989030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.990910 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.990888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:55.992612 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:55.992588 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f" path="/var/lib/kubelet/pods/9c5e9fa1-b3a5-4b78-8c30-74f1f3d5c52f/volumes" Sep 17 19:18:56.000962 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:56.000943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg77z\" (UniqueName: \"kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:56.151785 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:56.151764 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:18:56.426592 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:56.426565 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:18:56.428309 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:18:56.428284 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf6bd0a_eabb_4114_9a3e_a02ba59b70f7.slice/crio-21c6713fe7d3d100dbdcfb4120c95f520626b53c6011350a98c69023910ebbb2 WatchSource:0}: Error finding container 21c6713fe7d3d100dbdcfb4120c95f520626b53c6011350a98c69023910ebbb2: Status 404 returned error can't find the container with id 21c6713fe7d3d100dbdcfb4120c95f520626b53c6011350a98c69023910ebbb2 Sep 17 19:18:57.422214 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:57.422174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerStarted","Data":"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04"} Sep 17 19:18:57.422399 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:18:57.422239 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerStarted","Data":"21c6713fe7d3d100dbdcfb4120c95f520626b53c6011350a98c69023910ebbb2"} Sep 17 19:19:00.435911 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:00.435882 2559 generic.go:356] "Generic (PLEG): container finished" podID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerID="3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04" exitCode=0 Sep 17 19:19:00.436355 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:00.435919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerDied","Data":"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04"} Sep 17 19:19:01.442071 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.442034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerStarted","Data":"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548"} Sep 17 19:19:01.442480 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.442101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerStarted","Data":"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601"} Sep 17 19:19:01.442480 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.442401 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:19:01.442577 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.442555 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:19:01.443865 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.443841 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:01.473027 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:01.472971 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podStartSLOduration=2.914033918 podStartE2EDuration="6.472952232s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.558918314s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:01.471104097 +0000 UTC m=+2993.459720349" watchObservedRunningTime="2026-09-17 19:19:01.472952232 +0000 UTC m=+2993.461568513" Sep 17 19:19:02.446894 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:02.446855 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:07.451931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:07.451894 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:19:07.452608 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:07.452577 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:10.150519 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:10.150490 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:19:10.154246 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:10.154222 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:19:10.158969 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:10.158944 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:19:10.162249 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:10.162229 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:19:17.452753 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:17.452693 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:27.452551 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:27.452501 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:37.453313 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:37.453270 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:47.452796 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:47.452747 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:56.327592 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.327506 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:19:56.330977 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.330960 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.340012 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.339990 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-7e891e-dockercfg-52zww\"" Sep 17 19:19:56.340127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.339995 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-7e891e-predictor-serving-cert\"" Sep 17 19:19:56.340127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.340034 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-7e891e-kube-rbac-proxy-sar-config\"" Sep 17 19:19:56.340127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.340019 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-7e891e\"" Sep 17 19:19:56.358060 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.358034 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:19:56.392757 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.392718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgkq\" (UniqueName: \"kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.392757 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.392762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.393008 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.392801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.393008 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.392830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.393008 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.392851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494122 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494348 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgkq\" (UniqueName: \"kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494348 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494186 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494348 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494348 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.494805 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.494769 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.495032 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.495011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.495171 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.495055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.496835 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.496808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.504217 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.504186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgkq\" (UniqueName: \"kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq\") pod \"isvc-secondary-7e891e-predictor-57c9748f99-pln4t\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.640503 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.640438 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:19:56.942673 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:56.940493 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:19:56.945895 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:19:56.945865 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27e0010_c523_41fc_b8f7_9ebf7a8d8e6e.slice/crio-a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280 WatchSource:0}: Error finding container a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280: Status 404 returned error can't find the container with id a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280 Sep 17 19:19:57.453032 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:57.452987 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:19:57.691863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:57.691822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerStarted","Data":"00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c"} Sep 17 19:19:57.692044 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:19:57.691891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerStarted","Data":"a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280"} Sep 17 19:20:01.710904 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:01.710875 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/0.log" Sep 17 19:20:01.711410 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:01.710920 2559 generic.go:356] "Generic (PLEG): container finished" podID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerID="00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c" exitCode=1 Sep 17 19:20:01.711410 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:01.710989 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerDied","Data":"00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c"} Sep 17 19:20:02.716978 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:02.716948 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/0.log" Sep 17 19:20:02.717391 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:02.717025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerStarted","Data":"adafb993b5922522d049a10832ba14ae25fdd80a300a6e9f9fd4fee5d415a01b"} Sep 17 19:20:05.731060 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.731030 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/1.log" Sep 17 19:20:05.731578 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.731559 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/0.log" Sep 17 19:20:05.731671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.731605 2559 generic.go:356] "Generic (PLEG): container finished" podID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerID="adafb993b5922522d049a10832ba14ae25fdd80a300a6e9f9fd4fee5d415a01b" exitCode=1 Sep 17 19:20:05.731724 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.731682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerDied","Data":"adafb993b5922522d049a10832ba14ae25fdd80a300a6e9f9fd4fee5d415a01b"} Sep 17 19:20:05.731771 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.731750 2559 scope.go:117] "RemoveContainer" containerID="00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c" Sep 17 19:20:05.732263 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.732243 2559 scope.go:117] "RemoveContainer" containerID="00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c" Sep 17 19:20:05.743032 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:05.743008 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-7e891e-predictor-57c9748f99-pln4t_kserve-ci-e2e-test(c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e)\"" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" Sep 17 19:20:05.743226 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:05.743192 2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_kserve-ci-e2e-test_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e_0 in pod sandbox a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280 from index: no such id: '00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c'" containerID="00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c" Sep 17 19:20:05.743275 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:05.743238 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_kserve-ci-e2e-test_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e_0 in pod sandbox a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280 from index: no such id: '00deb7129d8ab09d62ae380e9b564f3f30e221d7cc82ff5866859b1c64a9313c'" Sep 17 19:20:06.737467 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:06.737414 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/1.log" Sep 17 19:20:07.452940 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:07.452899 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:20:12.219132 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.219087 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:20:12.361351 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.361326 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/1.log" Sep 17 19:20:12.363638 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.363622 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:20:12.430493 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430436 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls\") pod \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " Sep 17 19:20:12.430671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430511 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location\") pod \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " Sep 17 19:20:12.430671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config\") pod \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " Sep 17 19:20:12.430671 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430656 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txgkq\" (UniqueName: \"kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq\") pod \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " Sep 17 19:20:12.430787 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430687 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert\") pod \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\" (UID: \"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e\") " Sep 17 19:20:12.430833 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430784 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" (UID: "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:20:12.430957 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" (UID: "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:20:12.431021 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.430959 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:12.431113 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.431091 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" (UID: "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:20:12.432801 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.432772 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq" (OuterVolumeSpecName: "kube-api-access-txgkq") pod "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" (UID: "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e"). InnerVolumeSpecName "kube-api-access-txgkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:20:12.432801 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.432783 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" (UID: "c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:20:12.531430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.531340 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:12.531430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.531371 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgkq\" (UniqueName: \"kubernetes.io/projected/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-kube-api-access-txgkq\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:12.531430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.531381 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:12.531430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.531390 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:12.764227 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.764189 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7e891e-predictor-57c9748f99-pln4t_c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/storage-initializer/1.log" Sep 17 19:20:12.764401 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.764277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" event={"ID":"c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e","Type":"ContainerDied","Data":"a3382ea6c689d4694c2df4b360255ce395e2365cdbdc364dc71dcf5d36e5d280"} Sep 17 19:20:12.764401 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.764302 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t" Sep 17 19:20:12.764401 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.764336 2559 scope.go:117] "RemoveContainer" containerID="adafb993b5922522d049a10832ba14ae25fdd80a300a6e9f9fd4fee5d415a01b" Sep 17 19:20:12.814434 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.814345 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:20:12.821603 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:12.821572 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7e891e-predictor-57c9748f99-pln4t"] Sep 17 19:20:13.993176 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:13.993140 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" path="/var/lib/kubelet/pods/c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e/volumes" Sep 17 19:20:17.453429 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:17.453390 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:20:27.453326 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:27.453283 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:20:27.987931 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:27.987880 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused" Sep 17 19:20:37.993050 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:37.993018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:20:48.130730 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:48.130694 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:20:48.131143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:48.130957 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" containerID="cri-o://49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601" gracePeriod=30 Sep 17 19:20:48.131143 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:48.131011 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kube-rbac-proxy" containerID="cri-o://196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548" gracePeriod=30 Sep 17 19:20:48.168986 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:48.168954 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf6bd0a_eabb_4114_9a3e_a02ba59b70f7.slice/crio-conmon-196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548.scope\": RecentStats: unable to find data in memory cache]" Sep 17 19:20:48.917039 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:48.917006 2559 generic.go:356] "Generic (PLEG): container finished" podID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerID="196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548" exitCode=2 Sep 17 19:20:48.917228 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:48.917080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerDied","Data":"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548"} Sep 17 19:20:52.448127 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.448077 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused" Sep 17 19:20:52.679681 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.679656 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:20:52.764262 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764159 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location\") pod \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " Sep 17 19:20:52.764262 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert\") pod \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " Sep 17 19:20:52.764262 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764252 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg77z\" (UniqueName: \"kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z\") pod \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " Sep 17 19:20:52.764580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764286 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls\") pod \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " Sep 17 19:20:52.764580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config\") pod \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\" (UID: \"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7\") " Sep 17 19:20:52.764580 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" (UID: "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:20:52.764735 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764708 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" (UID: "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:20:52.764798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764716 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kserve-provision-location\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:52.764873 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.764851 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" (UID: "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 19:20:52.766499 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.766478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z" (OuterVolumeSpecName: "kube-api-access-lg77z") pod "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" (UID: "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7"). InnerVolumeSpecName "kube-api-access-lg77z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:20:52.766499 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.766490 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" (UID: "baf6bd0a-eabb-4114-9a3e-a02ba59b70f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 19:20:52.865222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.865189 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-cabundle-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:52.865222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.865215 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg77z\" (UniqueName: \"kubernetes.io/projected/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-api-access-lg77z\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:52.865222 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.865224 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-proxy-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:52.865477 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.865234 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:20:52.935942 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.935904 2559 generic.go:356] "Generic (PLEG): container finished" podID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerID="49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601" exitCode=0 Sep 17 19:20:52.936118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.935960 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerDied","Data":"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601"} Sep 17 19:20:52.936118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.935971 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" Sep 17 19:20:52.936118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.936059 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz" event={"ID":"baf6bd0a-eabb-4114-9a3e-a02ba59b70f7","Type":"ContainerDied","Data":"21c6713fe7d3d100dbdcfb4120c95f520626b53c6011350a98c69023910ebbb2"} Sep 17 19:20:52.936118 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.936101 2559 scope.go:117] "RemoveContainer" containerID="196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548" Sep 17 19:20:52.944591 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.944549 2559 scope.go:117] "RemoveContainer" containerID="49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601" Sep 17 19:20:52.952027 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.952004 2559 scope.go:117] "RemoveContainer" containerID="3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04" Sep 17 19:20:52.959282 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.959262 2559 scope.go:117] "RemoveContainer" containerID="196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548" Sep 17 19:20:52.959556 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:52.959535 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548\": container with ID starting with 196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548 not found: ID does not exist" containerID="196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548" Sep 17 19:20:52.959627 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.959565 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548"} err="failed to get container status \"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548\": rpc error: code = NotFound desc = could not find container \"196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548\": container with ID starting with 196d9ebdfb7b21e378c3d37a983bdd6747e5513e06fe85eec1a0e6f15fb8d548 not found: ID does not exist" Sep 17 19:20:52.959627 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.959582 2559 scope.go:117] "RemoveContainer" containerID="49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601" Sep 17 19:20:52.959822 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:52.959807 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601\": container with ID starting with 49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601 not found: ID does not exist" containerID="49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601" Sep 17 19:20:52.959863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.959826 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601"} err="failed to get container status \"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601\": rpc error: code = NotFound desc = could not find container \"49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601\": container with ID starting with 49c5d78d25b6339e83ab1f5b816dc15171d4d93f81c1e6896409d530070cf601 not found: ID does not exist" Sep 17 19:20:52.959863 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.959839 2559 scope.go:117] "RemoveContainer" containerID="3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04" Sep 17 19:20:52.960022 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:20:52.960006 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04\": container with ID starting with 3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04 not found: ID does not exist" containerID="3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04" Sep 17 19:20:52.960065 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.960026 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04"} err="failed to get container status \"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04\": rpc error: code = NotFound desc = could not find container \"3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04\": container with ID starting with 3f9db890656f072d195a920782e5390cb7f40f9697b71a0cedee9858022e1a04 not found: ID does not exist" Sep 17 19:20:52.965574 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.965547 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:20:52.967781 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:52.967759 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-xkclz"] Sep 17 19:20:53.991934 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:53.991903 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" path="/var/lib/kubelet/pods/baf6bd0a-eabb-4114-9a3e-a02ba59b70f7/volumes" Sep 17 19:20:54.150458 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150420 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkf64/must-gather-6qkcv"] Sep 17 19:20:54.150734 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150723 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150736 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150748 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150754 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150760 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150766 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150780 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150785 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="storage-initializer" Sep 17 19:20:54.150794 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150797 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kube-rbac-proxy" Sep 17 19:20:54.151048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150802 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kube-rbac-proxy" Sep 17 19:20:54.151048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150851 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kube-rbac-proxy" Sep 17 19:20:54.151048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150862 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.151048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150868 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e0010-c523-41fc-b8f7-9ebf7a8d8e6e" containerName="storage-initializer" Sep 17 19:20:54.151048 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.150874 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf6bd0a-eabb-4114-9a3e-a02ba59b70f7" containerName="kserve-container" Sep 17 19:20:54.155619 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.155602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.158488 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.158470 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkf64\"/\"kube-root-ca.crt\"" Sep 17 19:20:54.158831 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.158809 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkf64\"/\"openshift-service-ca.crt\"" Sep 17 19:20:54.173687 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.173653 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkf64/must-gather-6qkcv"] Sep 17 19:20:54.175469 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.175430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.175592 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.175501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.276798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.276699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.276798 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.276756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.277130 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.277109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.288779 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.288751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7\") pod \"must-gather-6qkcv\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.481787 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.481747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:20:54.754939 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.754840 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkf64/must-gather-6qkcv"] Sep 17 19:20:54.759984 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:20:54.759915 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33210880_dd70_48b2_8919_b5dde46c7c1d.slice/crio-3e295e02c04712519759573e27d5435763b63cc504a31138c42383c496c4251d WatchSource:0}: Error finding container 3e295e02c04712519759573e27d5435763b63cc504a31138c42383c496c4251d: Status 404 returned error can't find the container with id 3e295e02c04712519759573e27d5435763b63cc504a31138c42383c496c4251d Sep 17 19:20:54.761772 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.761753 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 19:20:54.945933 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:20:54.945899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkf64/must-gather-6qkcv" event={"ID":"33210880-dd70-48b2-8919-b5dde46c7c1d","Type":"ContainerStarted","Data":"3e295e02c04712519759573e27d5435763b63cc504a31138c42383c496c4251d"} Sep 17 19:21:01.986142 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:01.986103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkf64/must-gather-6qkcv" event={"ID":"33210880-dd70-48b2-8919-b5dde46c7c1d","Type":"ContainerStarted","Data":"441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2"} Sep 17 19:21:01.986541 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:01.986153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkf64/must-gather-6qkcv" event={"ID":"33210880-dd70-48b2-8919-b5dde46c7c1d","Type":"ContainerStarted","Data":"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d"} Sep 17 19:21:02.006946 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:02.006888 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bkf64/must-gather-6qkcv" podStartSLOduration=1.8294409360000001 podStartE2EDuration="8.006868668s" totalImagesPullingTime="6.177427732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:02.005720621 +0000 UTC m=+3113.994336873" watchObservedRunningTime="2026-09-17 19:21:02.006868668 +0000 UTC m=+3113.995484918" Sep 17 19:21:33.140430 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:33.140398 2559 generic.go:356] "Generic (PLEG): container finished" podID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerID="7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d" exitCode=0 Sep 17 19:21:33.140890 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:33.140482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkf64/must-gather-6qkcv" event={"ID":"33210880-dd70-48b2-8919-b5dde46c7c1d","Type":"ContainerDied","Data":"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d"} Sep 17 19:21:33.140890 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:33.140813 2559 scope.go:117] "RemoveContainer" containerID="7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d" Sep 17 19:21:33.254105 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:33.254071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkf64_must-gather-6qkcv_33210880-dd70-48b2-8919-b5dde46c7c1d/gather/0.log" Sep 17 19:21:37.555288 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:37.555256 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lx2z8_ea05b683-b14d-4b78-b315-f3daaed894e0/global-pull-secret-syncer/0.log" Sep 17 19:21:37.775758 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:37.775723 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-csc94_2fe05e5e-1aff-454a-9409-585a30fc2aae/konnectivity-agent/0.log" Sep 17 19:21:37.891385 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:37.891359 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-9.ec2.internal_cb7ea012701865f5588b4ef5f36a7e27/haproxy/0.log" Sep 17 19:21:38.835813 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:38.835772 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bkf64/must-gather-6qkcv"] Sep 17 19:21:38.836241 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:38.835970 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bkf64/must-gather-6qkcv" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="copy" containerID="cri-o://441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2" gracePeriod=2 Sep 17 19:21:38.841112 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:38.841085 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bkf64/must-gather-6qkcv"] Sep 17 19:21:39.066960 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.066935 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkf64_must-gather-6qkcv_33210880-dd70-48b2-8919-b5dde46c7c1d/copy/0.log" Sep 17 19:21:39.068596 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.068579 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.071305 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.071280 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6qkcv\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkf64\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.073544 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.073529 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output\") pod \"33210880-dd70-48b2-8919-b5dde46c7c1d\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " Sep 17 19:21:39.073636 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.073616 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7\") pod \"33210880-dd70-48b2-8919-b5dde46c7c1d\" (UID: \"33210880-dd70-48b2-8919-b5dde46c7c1d\") " Sep 17 19:21:39.075033 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.075009 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "33210880-dd70-48b2-8919-b5dde46c7c1d" (UID: "33210880-dd70-48b2-8919-b5dde46c7c1d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 19:21:39.075740 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.075718 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7" (OuterVolumeSpecName: "kube-api-access-6lsb7") pod "33210880-dd70-48b2-8919-b5dde46c7c1d" (UID: "33210880-dd70-48b2-8919-b5dde46c7c1d"). InnerVolumeSpecName "kube-api-access-6lsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 19:21:39.165913 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.165887 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkf64_must-gather-6qkcv_33210880-dd70-48b2-8919-b5dde46c7c1d/copy/0.log" Sep 17 19:21:39.166380 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.166353 2559 generic.go:356] "Generic (PLEG): container finished" podID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerID="441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2" exitCode=143 Sep 17 19:21:39.166509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.166412 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.166509 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.166476 2559 scope.go:117] "RemoveContainer" containerID="441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2" Sep 17 19:21:39.169186 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.169151 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6qkcv\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkf64\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.174540 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.174467 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/33210880-dd70-48b2-8919-b5dde46c7c1d-kube-api-access-6lsb7\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:21:39.174540 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.174493 2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33210880-dd70-48b2-8919-b5dde46c7c1d-must-gather-output\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 17 19:21:39.174637 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.174590 2559 scope.go:117] "RemoveContainer" containerID="7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d" Sep 17 19:21:39.177267 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.177236 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6qkcv\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkf64\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.187266 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.187247 2559 scope.go:117] "RemoveContainer" containerID="441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2" Sep 17 19:21:39.187650 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.187630 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2\": container with ID starting with 441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2 not found: ID does not exist" containerID="441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2" Sep 17 19:21:39.187713 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.187660 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2"} err="failed to get container status \"441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2\": rpc error: code = NotFound desc = could not find container \"441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2\": container with ID starting with 441e28740d115923ebecacc37c312dae80fe112739cb7aac30055654f520b1e2 not found: ID does not exist" Sep 17 19:21:39.187713 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.187681 2559 scope.go:117] "RemoveContainer" containerID="7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d" Sep 17 19:21:39.187947 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.187926 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d\": container with ID starting with 7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d not found: ID does not exist" containerID="7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d" Sep 17 19:21:39.188019 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.187957 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d"} err="failed to get container status \"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d\": rpc error: code = NotFound desc = could not find container \"7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d\": container with ID starting with 7bd98c802317337edfd82f9fa3dbb5be17198387c516752d599cd6a1b465081d not found: ID does not exist" Sep 17 19:21:39.991898 ip-10-0-132-9 kubenswrapper[2559]: E0917 19:21:39.991860 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6qkcv\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkf64\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" pod="openshift-must-gather-bkf64/must-gather-6qkcv" Sep 17 19:21:39.992593 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:39.992570 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" path="/var/lib/kubelet/pods/33210880-dd70-48b2-8919-b5dde46c7c1d/volumes" Sep 17 19:21:41.160899 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.160865 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-4hjzm_3780a308-335e-49bc-9739-ffee7515f558/cluster-monitoring-operator/0.log" Sep 17 19:21:41.584998 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.584968 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcnv7_4e5fecc6-e44f-4940-8f36-7e0ddd531b91/node-exporter/0.log" Sep 17 19:21:41.614201 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.614169 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcnv7_4e5fecc6-e44f-4940-8f36-7e0ddd531b91/kube-rbac-proxy/0.log" Sep 17 19:21:41.639179 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.639150 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcnv7_4e5fecc6-e44f-4940-8f36-7e0ddd531b91/init-textfile/0.log" Sep 17 19:21:41.779602 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.779524 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/prometheus/0.log" Sep 17 19:21:41.800871 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.800847 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/config-reloader/0.log" Sep 17 19:21:41.827386 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.827357 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/thanos-sidecar/0.log" Sep 17 19:21:41.855533 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.855514 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/kube-rbac-proxy-web/0.log" Sep 17 19:21:41.880741 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.880719 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/kube-rbac-proxy/0.log" Sep 17 19:21:41.909298 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.909274 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/kube-rbac-proxy-thanos/0.log" Sep 17 19:21:41.932640 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:41.932616 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_524c6f38-1f7f-41a7-82fc-621174e93c6f/init-config-reloader/0.log" Sep 17 19:21:42.017912 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:42.017888 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-lgtcm_10471f87-84c2-49bf-bdb1-e13094bb018c/prometheus-operator-admission-webhook/0.log" Sep 17 19:21:43.631629 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:43.631602 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-9nf5r_7736fba4-d5d5-4bb2-b963-28ed95471ce9/networking-console-plugin/0.log" Sep 17 19:21:44.192051 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:44.192021 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/1.log" Sep 17 19:21:44.196346 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:44.196327 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fw9dt_4efa5014-c43a-41f9-937a-f83becc7cd54/console-operator/2.log" Sep 17 19:21:44.683277 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:44.683245 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-jfk46_5e625bb8-3370-4be6-881a-e29b096e30d6/download-server/0.log" Sep 17 19:21:45.210977 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.210944 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-z87pf_318bf57e-5b23-476e-9f2d-a9f9730425c0/volume-data-source-validator/0.log" Sep 17 19:21:45.467019 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.466938 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk"] Sep 17 19:21:45.467242 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467231 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="copy" Sep 17 19:21:45.467285 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467244 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="copy" Sep 17 19:21:45.467285 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467254 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="gather" Sep 17 19:21:45.467285 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467260 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="gather" Sep 17 19:21:45.467387 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467323 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="gather" Sep 17 19:21:45.467387 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.467331 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33210880-dd70-48b2-8919-b5dde46c7c1d" containerName="copy" Sep 17 19:21:45.470365 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.470345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.473205 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.473183 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hmqj2\"/\"default-dockercfg-vrw5k\"" Sep 17 19:21:45.473343 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.473186 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqj2\"/\"openshift-service-ca.crt\"" Sep 17 19:21:45.474185 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.474171 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hmqj2\"/\"kube-root-ca.crt\"" Sep 17 19:21:45.479731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.479706 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk"] Sep 17 19:21:45.523367 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.523326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-proc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.523588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.523381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-sys\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.523588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.523400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ztc\" (UniqueName: \"kubernetes.io/projected/aaa4db1d-1fa5-4980-a463-bd33b4b77717-kube-api-access-d4ztc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.523588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.523437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-podres\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.523588 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.523477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-lib-modules\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.624846 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-proc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-sys\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ztc\" (UniqueName: \"kubernetes.io/projected/aaa4db1d-1fa5-4980-a463-bd33b4b77717-kube-api-access-d4ztc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-podres\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-lib-modules\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-proc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.624948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-sys\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625059 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.625052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-podres\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.625391 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.625118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaa4db1d-1fa5-4980-a463-bd33b4b77717-lib-modules\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.633842 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.633808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ztc\" (UniqueName: \"kubernetes.io/projected/aaa4db1d-1fa5-4980-a463-bd33b4b77717-kube-api-access-d4ztc\") pod \"perf-node-gather-daemonset-5dvdk\" (UID: \"aaa4db1d-1fa5-4980-a463-bd33b4b77717\") " pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:45.781084 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:45.780985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:46.054551 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.054524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk"] Sep 17 19:21:46.056675 ip-10-0-132-9 kubenswrapper[2559]: W0917 19:21:46.056650 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaaa4db1d_1fa5_4980_a463_bd33b4b77717.slice/crio-b2a5578cf3b4a1263d8588b69699e029728f1fda89741c008ce688ff39e4b8bb WatchSource:0}: Error finding container b2a5578cf3b4a1263d8588b69699e029728f1fda89741c008ce688ff39e4b8bb: Status 404 returned error can't find the container with id b2a5578cf3b4a1263d8588b69699e029728f1fda89741c008ce688ff39e4b8bb Sep 17 19:21:46.188175 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.188142 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crrcl_1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e/dns/0.log" Sep 17 19:21:46.197152 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.197116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" event={"ID":"aaa4db1d-1fa5-4980-a463-bd33b4b77717","Type":"ContainerStarted","Data":"ee2d45c40ac85556fdb51a532fa4494df41b11e3b7ae562bedc15bf5d52d64d3"} Sep 17 19:21:46.197364 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.197174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" event={"ID":"aaa4db1d-1fa5-4980-a463-bd33b4b77717","Type":"ContainerStarted","Data":"b2a5578cf3b4a1263d8588b69699e029728f1fda89741c008ce688ff39e4b8bb"} Sep 17 19:21:46.197364 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.197203 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:46.218215 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.218183 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crrcl_1ca44f7c-8d5b-4a4e-b842-d17e698d2c5e/kube-rbac-proxy/0.log" Sep 17 19:21:46.218589 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.218556 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" podStartSLOduration=1.218542958 podStartE2EDuration="1.218542958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:46.216536199 +0000 UTC m=+3158.205152448" watchObservedRunningTime="2026-09-17 19:21:46.218542958 +0000 UTC m=+3158.207159206" Sep 17 19:21:46.354539 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.354419 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j2lmq_4583ef38-ee5c-4056-9acd-00782d3adf4b/dns-node-resolver/0.log" Sep 17 19:21:46.911620 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:46.911591 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-29dvp_f5011467-2009-4336-ac87-cd9bf72b1881/node-ca/0.log" Sep 17 19:21:47.926077 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:47.926050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7748c668d8-2g2jw_e6d0f38d-71ce-4a7e-9d11-31694639677f/router/0.log" Sep 17 19:21:48.261145 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.261021 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn5p4_a2a823a4-9218-4703-ba05-6aa9d41b183b/serve-healthcheck-canary/0.log" Sep 17 19:21:48.665474 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.665429 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-8kmsh_38512eae-3e24-4fd5-992b-2861f70dbedc/insights-operator/0.log" Sep 17 19:21:48.665681 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.665633 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-8kmsh_38512eae-3e24-4fd5-992b-2861f70dbedc/insights-operator/1.log" Sep 17 19:21:48.755230 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.755204 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rvsm_a2768270-206e-4611-9ca1-b9937f1dad79/kube-rbac-proxy/0.log" Sep 17 19:21:48.775835 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.775810 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rvsm_a2768270-206e-4611-9ca1-b9937f1dad79/exporter/0.log" Sep 17 19:21:48.797257 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:48.797231 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rvsm_a2768270-206e-4611-9ca1-b9937f1dad79/extractor/0.log" Sep 17 19:21:51.666095 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.666057 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6z876_6a6ade4b-fc47-49d6-a4dc-9b113eef1d81/s3-init/0.log" Sep 17 19:21:51.685832 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.685804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6z876_6a6ade4b-fc47-49d6-a4dc-9b113eef1d81/download-hf-model/0.log" Sep 17 19:21:51.714496 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.714442 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6svb5_459fb52c-abc0-41db-b840-a0bd71c48c7a/s3-tls-init-custom/0.log" Sep 17 19:21:51.732411 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.732382 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6svb5_459fb52c-abc0-41db-b840-a0bd71c48c7a/download-hf-model/0.log" Sep 17 19:21:51.789277 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.789239 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-c2rkn_7a6e017e-fb4e-4d60-aff2-cd42d7df536f/s3-tls-init-serving/0.log" Sep 17 19:21:51.806168 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:51.806140 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-c2rkn_7a6e017e-fb4e-4d60-aff2-cd42d7df536f/download-hf-model/0.log" Sep 17 19:21:52.212013 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:52.211981 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hmqj2/perf-node-gather-daemonset-5dvdk" Sep 17 19:21:58.325207 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:58.325166 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-m6w5d_acf34cb8-1ea1-401d-877f-0e11ead44a7d/kube-storage-version-migrator-operator/1.log" Sep 17 19:21:58.326516 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:58.326497 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-m6w5d_acf34cb8-1ea1-401d-877f-0e11ead44a7d/kube-storage-version-migrator-operator/0.log" Sep 17 19:21:59.342710 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.342680 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/kube-multus-additional-cni-plugins/0.log" Sep 17 19:21:59.366726 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.366698 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/egress-router-binary-copy/0.log" Sep 17 19:21:59.388850 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.388823 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/cni-plugins/0.log" Sep 17 19:21:59.415332 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.415307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/bond-cni-plugin/0.log" Sep 17 19:21:59.438724 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.438692 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/routeoverride-cni/0.log" Sep 17 19:21:59.465690 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.465665 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/whereabouts-cni-bincopy/0.log" Sep 17 19:21:59.490802 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.490778 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g9dfm_1c2db46f-3b5d-43db-a562-7eedafad93ce/whereabouts-cni/0.log" Sep 17 19:21:59.783950 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.783907 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g72sf_dd4073a8-521e-483a-abb9-0506b859d73c/kube-multus/0.log" Sep 17 19:21:59.844125 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.844094 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jq4tl_e08ab8d9-eb73-4356-b19c-3250af393478/network-metrics-daemon/0.log" Sep 17 19:21:59.865334 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:21:59.865304 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jq4tl_e08ab8d9-eb73-4356-b19c-3250af393478/kube-rbac-proxy/0.log" Sep 17 19:22:01.070246 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.070219 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-controller/0.log" Sep 17 19:22:01.131482 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.131438 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/1.log" Sep 17 19:22:01.131482 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.131475 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovn-acl-logging/0.log" Sep 17 19:22:01.179754 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.179722 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/kube-rbac-proxy-node/0.log" Sep 17 19:22:01.215468 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.215425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 19:22:01.242486 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.242461 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/northd/0.log" Sep 17 19:22:01.267651 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.267623 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/nbdb/0.log" Sep 17 19:22:01.290807 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.290776 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/sbdb/0.log" Sep 17 19:22:01.335478 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:01.335392 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gbgqb_d409999b-ee91-4485-868b-4162e1b49553/ovnkube-controller/0.log" Sep 17 19:22:02.538732 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:02.538696 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-28z94_397b6ef1-7013-42ce-86e6-9f50acfaaf93/check-endpoints/0.log" Sep 17 19:22:02.617731 ip-10-0-132-9 kubenswrapper[2559]: I0917 19:22:02.617698 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lx76w_ae36918f-41a8-4fd0-abdd-42d3b478f0f3/network-check-target-container/0.log"