Jul 28 15:32:59.827524 ip-10-0-136-8 systemd[1]: Starting Kubernetes Kubelet... Jul 28 15:33:02.091801 ip-10-0-136-8 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:02.091801 ip-10-0-136-8 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 15:33:02.091801 ip-10-0-136-8 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:02.091801 ip-10-0-136-8 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 15:33:02.091801 ip-10-0-136-8 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:02.092486 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.091863 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 15:33:02.097079 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097052 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.097079 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097070 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.097079 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097074 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.097079 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097080 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.097079 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097085 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097088 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097091 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097095 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097098 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097101 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097104 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097106 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097109 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097112 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097114 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097117 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097120 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097122 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097125 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097128 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097130 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097133 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097136 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097139 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.097287 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097141 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097146 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097149 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097152 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097155 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097158 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097165 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097168 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097171 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097174 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097177 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097180 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097182 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097185 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097189 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097192 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097194 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097197 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097200 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.097810 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097202 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097206 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097208 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097211 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097215 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097218 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097221 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097225 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097227 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097230 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097233 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097235 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097238 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097241 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097243 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097247 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097249 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097252 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097254 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097258 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.098294 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097260 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097263 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097265 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097268 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097271 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097274 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097277 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097280 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097286 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097290 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097293 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097296 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097298 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097301 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097304 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097306 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097309 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097312 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097315 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097317 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.098788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097320 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097323 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097326 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097328 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097331 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097333 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097336 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097338 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097341 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097344 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097346 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097349 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097352 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097355 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097359 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.097361 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098085 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098092 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098095 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098098 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.099305 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098101 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098103 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098106 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098109 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098111 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098114 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098117 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098119 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098122 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098125 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098128 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098130 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098133 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098136 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098139 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098141 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098144 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098147 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098149 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098152 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.099800 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098155 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098158 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098160 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098163 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098166 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098168 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098172 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098174 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098177 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098180 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098182 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098185 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098187 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098190 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098192 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098195 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098197 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098200 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098202 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098205 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.100319 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098207 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098210 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098212 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098243 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098247 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098249 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098254 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098258 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098261 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098265 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098268 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098271 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098274 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098278 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098280 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098284 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098287 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098290 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098293 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.100820 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098296 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098299 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098302 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098305 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098308 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098311 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098313 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098316 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098318 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098321 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098323 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098326 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098328 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098331 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098333 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098336 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098339 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098341 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098344 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.101303 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098347 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098349 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098353 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098357 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098359 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098362 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098364 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098367 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098370 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098372 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098375 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098378 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098380 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098383 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098386 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098403 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098406 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098796 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098801 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.101788 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098804 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098806 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098811 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098815 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098826 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098829 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098832 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098835 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098838 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098841 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098844 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098846 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098849 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098852 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098856 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098859 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098861 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098864 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098866 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.102241 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098869 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098872 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098875 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098878 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098880 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098883 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098886 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098889 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098891 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098894 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098896 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098899 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098901 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098904 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098907 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098911 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098914 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098917 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098920 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098923 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.102712 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098925 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098928 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098930 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098933 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098935 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098938 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098941 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098943 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098946 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098949 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098951 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098954 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098956 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098959 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098962 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098964 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098967 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098970 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098973 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098976 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.103204 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098978 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098980 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098983 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098985 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098988 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098990 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098993 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098995 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.098998 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099001 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099003 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099005 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099008 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099010 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099013 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099015 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099018 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099020 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099023 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099025 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.103705 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099028 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099032 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099035 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099038 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099041 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099048 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099051 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099056 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099059 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099061 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099064 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099066 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099069 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099071 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099074 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099076 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099079 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099082 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099163 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099171 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 15:33:02.104207 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099177 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099181 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099187 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099190 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099194 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099199 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099202 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099205 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099208 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099211 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099215 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099217 2575 flags.go:64] FLAG: --cgroup-root="" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099220 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099223 2575 flags.go:64] FLAG: --client-ca-file="" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099227 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099230 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099235 2575 flags.go:64] FLAG: --cluster-domain="" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099239 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099242 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099246 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099250 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099254 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099257 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 15:33:02.104709 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099260 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099263 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099266 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099269 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099272 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099275 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099278 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099283 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099286 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099289 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099292 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099295 2575 flags.go:64] FLAG: --enable-server="true" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099297 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099302 2575 flags.go:64] FLAG: --event-burst="100" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099305 2575 flags.go:64] FLAG: --event-qps="50" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099308 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099311 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099314 2575 flags.go:64] FLAG: --eviction-hard="" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099318 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099321 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099324 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099327 2575 flags.go:64] FLAG: --eviction-soft="" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099330 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099333 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099336 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 15:33:02.105251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099339 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099343 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099346 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099350 2575 flags.go:64] FLAG: --feature-gates="" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099354 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099357 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099360 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099363 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099366 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099369 2575 flags.go:64] FLAG: --help="false" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099372 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099375 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099378 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099381 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099384 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099399 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099403 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099406 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099409 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099411 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099414 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099417 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099420 2575 flags.go:64] FLAG: --kube-reserved="" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099423 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 15:33:02.105905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099426 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099429 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099432 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099434 2575 flags.go:64] FLAG: --lock-file="" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099437 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099440 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099443 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099448 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099453 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099457 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099460 2575 flags.go:64] FLAG: --logging-format="text" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099464 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099467 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099470 2575 flags.go:64] FLAG: --manifest-url="" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099473 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099477 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099480 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099485 2575 flags.go:64] FLAG: --max-pods="110" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099487 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099491 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099494 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099497 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099500 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099502 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099505 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 15:33:02.106503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099513 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099516 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099519 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099522 2575 flags.go:64] FLAG: --pod-cidr="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099525 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099531 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099534 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099538 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099540 2575 flags.go:64] FLAG: --port="10250" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099543 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099546 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04191dbab3807f5d8" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099549 2575 flags.go:64] FLAG: --qos-reserved="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099553 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099555 2575 flags.go:64] FLAG: --register-node="true" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099558 2575 flags.go:64] FLAG: --register-with-taints="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099562 2575 flags.go:64] FLAG: --registry-burst="10" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099566 2575 flags.go:64] FLAG: --registry-qps="5" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099569 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099572 2575 flags.go:64] FLAG: --reserved-memory="" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099576 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099580 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099582 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099585 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099588 2575 flags.go:64] FLAG: --runonce="false" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099591 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 15:33:02.107117 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099594 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099596 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099599 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099602 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099605 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099608 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099611 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099614 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099617 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099620 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099622 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099626 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099628 2575 flags.go:64] FLAG: --system-cgroups="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099632 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099637 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099640 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099642 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099647 2575 flags.go:64] FLAG: --tls-min-version="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099649 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099652 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099655 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099658 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099660 2575 flags.go:64] FLAG: --v="2" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099665 2575 flags.go:64] FLAG: --version="false" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099670 2575 flags.go:64] FLAG: --vmodule="" Jul 28 15:33:02.107745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099674 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.099678 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099803 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099807 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099810 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099813 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099816 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099819 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099822 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099824 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099827 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099830 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099832 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099835 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099838 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099840 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099843 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099845 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099848 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099850 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.108361 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099853 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099855 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099858 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099860 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099863 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099865 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099868 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099871 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099873 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099876 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099881 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099883 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099887 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099891 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099893 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099896 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099898 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099901 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099904 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.109001 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099906 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099909 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099912 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099914 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099917 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099920 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099922 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099925 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099927 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099930 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099932 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099936 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099940 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099942 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099945 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099948 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099951 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099954 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099956 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099959 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.109497 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099961 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099964 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099966 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099970 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099973 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099975 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099980 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099983 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099985 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099988 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099990 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099993 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.099997 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100001 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100004 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100007 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100010 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100013 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100016 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100019 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.109979 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100022 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100025 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100027 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100030 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100033 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100035 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100038 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100041 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100043 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100046 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100048 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100051 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100055 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100058 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100061 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100065 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100068 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100071 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100075 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.110503 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100078 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100080 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.100083 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.100587 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.107060 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.107076 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.108560 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.108567 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108626 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108631 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108634 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108638 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108641 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108643 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108646 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.110968 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108649 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108652 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108654 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108657 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108660 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108662 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108665 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108667 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108670 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108672 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108675 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108678 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108681 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108684 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108686 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108689 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108691 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108694 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108697 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108699 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.111345 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108702 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108705 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108708 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108711 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108713 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108716 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108719 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108721 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108724 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108726 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108729 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108731 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108734 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108736 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108739 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108741 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108744 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108747 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108749 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108752 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.111834 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108754 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108757 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108759 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108762 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108765 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108768 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108771 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108773 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108776 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108778 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108783 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108787 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108790 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108793 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108796 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108799 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108802 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108805 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108808 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.112325 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108811 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108815 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108818 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108821 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108824 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108827 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108830 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108832 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108835 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108838 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108840 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108843 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108847 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108850 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108853 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108855 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108858 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108861 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108864 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.112807 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108867 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108870 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108874 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108878 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108881 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108884 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108887 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108890 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108892 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108895 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108897 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108899 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108902 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.108905 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.113268 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.108910 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109049 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109055 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109058 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109061 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109064 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109067 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109069 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109072 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109075 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109077 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109080 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109082 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109085 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109087 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109090 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109093 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109095 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109098 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:02.113633 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109101 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109103 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109105 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109108 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109111 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109114 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109116 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109119 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109121 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109124 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109127 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109129 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109132 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109135 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109137 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109140 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109142 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109145 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109147 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109150 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:02.114124 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109152 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109155 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109157 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109160 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109163 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109165 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109167 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109170 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109172 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109175 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109177 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109180 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109182 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109184 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109187 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109190 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109193 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109195 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109198 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109201 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:02.114619 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109203 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109206 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109208 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109210 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109213 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109216 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109218 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109221 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109223 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109226 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109228 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109231 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109235 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109238 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109241 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109244 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109246 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109249 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109252 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109255 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:02.115114 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109257 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109261 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109263 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109266 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109269 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109272 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109275 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109278 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109280 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109283 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109286 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109290 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109292 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109295 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109297 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109300 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109302 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109305 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109307 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109309 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:02.115646 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:02.109312 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.109317 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.109444 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.111527 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.112816 2575 server.go:1019] "Starting client certificate rotation" Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.112949 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.113732 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 15:33:02.116139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.115355 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 15:33:02.116814 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.116788 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 15:33:02.127838 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.127816 2575 log.go:25] "Validated CRI v1 runtime API" Jul 28 15:33:02.129874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.129859 2575 log.go:25] "Validated CRI v1 image API" Jul 28 15:33:02.132795 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.132769 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 15:33:02.136959 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.136932 2575 fs.go:135] Filesystem UUIDs: map[1972771a-3e3d-41cf-b659-22818c0f5510:/dev/nvme0n1p4 6a9b77ab-5c5e-4eaf-9176-8b5eeb49364d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 28 15:33:02.137041 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.136956 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 15:33:02.143047 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.142918 2575 manager.go:216] Machine: {Timestamp:2026-07-28 15:33:02.14166416 +0000 UTC m=+0.342511643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2962434 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28576e03fd07c87d28e9ea85737ca2 SystemUUID:ec28576e-03fd-07c8-7d28-e9ea85737ca2 BootID:e0fc61f6-efb5-4318-9066-0243f0e6c21f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:e6:e5:e9:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:e6:e5:e9:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:a4:2e:5d:ba:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 15:33:02.143047 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.143036 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 15:33:02.143232 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.143149 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 15:33:02.144963 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.144938 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 15:33:02.145115 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.144966 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 15:33:02.145758 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.145747 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 15:33:02.145758 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.145759 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 15:33:02.145853 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.145767 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 15:33:02.146420 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.146409 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 15:33:02.147310 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.147299 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 15:33:02.147941 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.147924 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 15:33:02.149260 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149248 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 28 15:33:02.149401 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149378 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 15:33:02.149446 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149420 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 28 15:33:02.149446 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149438 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 15:33:02.149536 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149457 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 15:33:02.149536 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149467 2575 kubelet.go:398] "Adding apiserver pod source" Jul 28 15:33:02.149536 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149477 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 15:33:02.149962 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149948 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 15:33:02.150011 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.149972 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 15:33:02.152360 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.152341 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 15:33:02.154278 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.154259 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 15:33:02.154355 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.154301 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 15:33:02.155824 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155809 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155829 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155838 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155846 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155858 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155866 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155878 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155886 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 15:33:02.155901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155901 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 15:33:02.156166 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155910 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 15:33:02.156166 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155922 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 15:33:02.156166 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.155930 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 15:33:02.156631 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.156620 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 15:33:02.156681 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.156633 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 15:33:02.159860 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.159837 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 15:33:02.159967 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.159919 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 15:33:02.159967 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.159923 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 15:33:02.160112 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.160099 2575 server.go:1268] "Started kubelet" Jul 28 15:33:02.160194 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.160166 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 15:33:02.160365 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.160277 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 15:33:02.160472 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.160460 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 15:33:02.160914 ip-10-0-136-8 systemd[1]: Started Kubernetes Kubelet. Jul 28 15:33:02.164564 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.164544 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 15:33:02.165771 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.165743 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 15:33:02.166331 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.166317 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 28 15:33:02.171480 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.170552 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-8.ec2.internal.18c67df25aedd00e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-8.ec2.internal,UID:ip-10-0-136-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-8.ec2.internal,},FirstTimestamp:2026-07-28 15:33:02.160072718 +0000 UTC m=+0.360920200,LastTimestamp:2026-07-28 15:33:02.160072718 +0000 UTC m=+0.360920200,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-8.ec2.internal,}" Jul 28 15:33:02.174264 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.174246 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 15:33:02.174355 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.174268 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 15:33:02.174355 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.174330 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.175032 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.174949 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 15:33:02.175032 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.174980 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 15:33:02.175032 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.174993 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 15:33:02.175203 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175117 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 28 15:33:02.175203 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175125 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 28 15:33:02.175203 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175170 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 15:33:02.175203 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175197 2575 factory.go:55] Registering systemd factory Jul 28 15:33:02.175385 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175207 2575 factory.go:223] Registration of the systemd container factory successfully Jul 28 15:33:02.175385 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175411 2575 factory.go:153] Registering CRI-O factory Jul 28 15:33:02.175531 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175445 2575 factory.go:223] Registration of the crio container factory successfully Jul 28 15:33:02.175531 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175468 2575 factory.go:103] Registering Raw factory Jul 28 15:33:02.175531 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.175481 2575 manager.go:1196] Started watching for new ooms in manager Jul 28 15:33:02.176627 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.176608 2575 manager.go:349] Starting recovery of all containers Jul 28 15:33:02.179811 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.179771 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 15:33:02.179999 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.179969 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 15:33:02.188062 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.188042 2575 manager.go:354] Recovery completed Jul 28 15:33:02.190089 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.190066 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wn9kp" Jul 28 15:33:02.192573 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.192560 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.194309 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194295 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.194368 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194322 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.194368 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194332 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.194934 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194917 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 15:33:02.194934 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194931 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 15:33:02.194934 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194932 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.195178 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.194946 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 28 15:33:02.196291 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196265 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.196379 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196299 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.196379 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196314 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.196379 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.196301 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-8.ec2.internal.18c67df25cf835f4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-8.ec2.internal,UID:ip-10-0-136-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-8.ec2.internal,},FirstTimestamp:2026-07-28 15:33:02.194308596 +0000 UTC m=+0.395156078,LastTimestamp:2026-07-28 15:33:02.194308596 +0000 UTC m=+0.395156078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-8.ec2.internal,}" Jul 28 15:33:02.196754 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196739 2575 policy_none.go:49] "None policy: Start" Jul 28 15:33:02.196829 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196763 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 15:33:02.196829 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.196774 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 15:33:02.197792 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.197781 2575 policy_none.go:47] "Start" Jul 28 15:33:02.204627 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.204612 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wn9kp" Jul 28 15:33:02.207068 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.207000 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-8.ec2.internal.18c67df25cf87def default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-8.ec2.internal,UID:ip-10-0-136-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-8.ec2.internal,},FirstTimestamp:2026-07-28 15:33:02.194327023 +0000 UTC m=+0.395174505,LastTimestamp:2026-07-28 15:33:02.194327023 +0000 UTC m=+0.395174505,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-8.ec2.internal,}" Jul 28 15:33:02.231166 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.230976 2575 manager.go:337] "Starting Device Plugin manager" Jul 28 15:33:02.231308 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.231193 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 15:33:02.231308 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231206 2575 server.go:87] "Starting device plugin registration server" Jul 28 15:33:02.231503 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231491 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 15:33:02.231540 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231505 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 15:33:02.231619 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231600 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 15:33:02.231708 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231693 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 15:33:02.231764 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.231711 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 15:33:02.232385 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.232364 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 15:33:02.232461 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.232440 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.332700 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.332608 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.333821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.333628 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.333821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.333666 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.333821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.333682 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.333821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.333728 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.341958 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.341937 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.342072 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.341964 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-8.ec2.internal\": node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.356783 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.356753 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.365682 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.365651 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 15:33:02.366929 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.366907 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 15:33:02.367029 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.366944 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 15:33:02.367029 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.366963 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 15:33:02.367029 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.366995 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 15:33:02.369343 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.369325 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:02.457220 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.457181 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.467368 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.467338 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal"] Jul 28 15:33:02.467466 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.467456 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.469145 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.469131 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.469227 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.469160 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.469227 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.469174 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.470477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.470465 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.470597 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.470581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.470649 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.470617 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.471247 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471226 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.471313 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471256 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.471313 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471267 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.471313 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471265 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.471313 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471294 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.471313 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.471305 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.472529 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.472516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.472586 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.472540 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:02.473192 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.473177 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:02.473277 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.473205 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:02.473277 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.473218 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:02.504201 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.504177 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-8.ec2.internal\" not found" node="ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.508590 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.508573 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-8.ec2.internal\" not found" node="ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.558202 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.558166 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.576602 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.576566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.576602 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.576598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82fc34b6950a9b1738a19b765063a610-config\") pod \"kube-apiserver-proxy-ip-10-0-136-8.ec2.internal\" (UID: \"82fc34b6950a9b1738a19b765063a610\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.576811 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.576617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.658974 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.658901 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.677274 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82fc34b6950a9b1738a19b765063a610-config\") pod \"kube-apiserver-proxy-ip-10-0-136-8.ec2.internal\" (UID: \"82fc34b6950a9b1738a19b765063a610\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.677274 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82fc34b6950a9b1738a19b765063a610-config\") pod \"kube-apiserver-proxy-ip-10-0-136-8.ec2.internal\" (UID: \"82fc34b6950a9b1738a19b765063a610\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.677427 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.677427 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.677427 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.677537 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.677426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ec366b83bd37199ae0b62f4af0e339-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal\" (UID: \"d0ec366b83bd37199ae0b62f4af0e339\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.759452 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.759422 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.806965 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.806929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.811888 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:02.811870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:02.860282 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.860246 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:02.960846 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:02.960769 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.061337 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.061299 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.086905 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.086882 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:03.112557 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.112515 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 15:33:03.113022 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.112696 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 15:33:03.113022 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.112727 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 15:33:03.161717 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.161692 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.209702 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.209661 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:28:02 +0000 UTC" deadline="2028-05-04 07:29:59.640068742 +0000 UTC" Jul 28 15:33:03.209702 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.209694 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15495h56m56.430377358s" Jul 28 15:33:03.261990 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.261957 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.323704 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:03.323654 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fc34b6950a9b1738a19b765063a610.slice/crio-ab4ab051afc40d0c8595745f69d5c42d4355d3f46495eff7528eddbae49b56c2 WatchSource:0}: Error finding container ab4ab051afc40d0c8595745f69d5c42d4355d3f46495eff7528eddbae49b56c2: Status 404 returned error can't find the container with id ab4ab051afc40d0c8595745f69d5c42d4355d3f46495eff7528eddbae49b56c2 Jul 28 15:33:03.323933 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:03.323915 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ec366b83bd37199ae0b62f4af0e339.slice/crio-c9c7b193669978cc2316ab351ef66833e205cecaabcbc2e1d6208d303b52bae6 WatchSource:0}: Error finding container c9c7b193669978cc2316ab351ef66833e205cecaabcbc2e1d6208d303b52bae6: Status 404 returned error can't find the container with id c9c7b193669978cc2316ab351ef66833e205cecaabcbc2e1d6208d303b52bae6 Jul 28 15:33:03.329557 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.329542 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:33:03.359048 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.359013 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:03.360246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.360220 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:03.360363 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.360257 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:03.360363 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.360267 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:03.362444 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.362425 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.370752 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.370712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" event={"ID":"82fc34b6950a9b1738a19b765063a610","Type":"ContainerStarted","Data":"ab4ab051afc40d0c8595745f69d5c42d4355d3f46495eff7528eddbae49b56c2"} Jul 28 15:33:03.371728 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.371702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" event={"ID":"d0ec366b83bd37199ae0b62f4af0e339","Type":"ContainerStarted","Data":"c9c7b193669978cc2316ab351ef66833e205cecaabcbc2e1d6208d303b52bae6"} Jul 28 15:33:03.386100 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.386069 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:03.463380 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.463343 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.564045 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:03.563999 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-8.ec2.internal\" not found" Jul 28 15:33:03.643311 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.643280 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:03.675111 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.675083 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" Jul 28 15:33:03.687717 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.687697 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 15:33:03.688860 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.688846 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" Jul 28 15:33:03.706592 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:03.706569 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 15:33:04.108511 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.108474 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:04.150462 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.150421 2575 apiserver.go:52] "Watching apiserver" Jul 28 15:33:04.159785 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.159758 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 15:33:04.160215 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.160187 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-mb5q9","openshift-network-operator/iptables-alerter-5c5fk","kube-system/konnectivity-agent-fr47v","openshift-image-registry/node-ca-sj6zz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal","openshift-multus/multus-6vpqn","openshift-multus/multus-additional-cni-plugins-nxmtr","openshift-multus/network-metrics-daemon-g965z","openshift-network-diagnostics/network-check-target-7j2v2","openshift-ovn-kubernetes/ovnkube-node-kgsxf","kube-system/global-pull-secret-syncer-wgwzd","kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn","openshift-cluster-node-tuning-operator/tuned-crt5d"] Jul 28 15:33:04.161865 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.161841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.163340 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.163318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.163482 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.163451 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:04.164482 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.164459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.165789 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.165760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.167562 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.167544 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.168084 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168064 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 15:33:04.168342 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168326 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 15:33:04.168458 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168361 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 15:33:04.168557 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168540 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.168615 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168582 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 15:33:04.168705 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168672 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 15:33:04.168705 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168684 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-628gw\"" Jul 28 15:33:04.169010 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.168989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.169450 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169431 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sjkrw\"" Jul 28 15:33:04.169636 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fxmrb\"" Jul 28 15:33:04.169636 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169624 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.169782 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169651 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.169782 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169727 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.169907 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169889 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.169959 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 15:33:04.169959 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.169918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cq7xp\"" Jul 28 15:33:04.170975 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.170356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:04.170975 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.170461 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:04.172057 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.171915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.172057 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.172037 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.172939 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.172918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.174434 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.174415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.175843 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.175233 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 15:33:04.176667 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.176643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.178544 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.178465 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.178631 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.178538 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:04.187377 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-socket-dir-parent\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.187494 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-hostroot\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-kubelet-config\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-systemd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-cnibin\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-socket-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.188072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.187957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-device-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.188550 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-kubelet\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188618 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-var-lib-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188674 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52a87494-68cd-4205-96cf-57bebf61dcdf-ovn-node-metrics-cert\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188674 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-netns\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.188674 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-konnectivity-ca\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-registration-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-ovn\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01e170ce-67ac-4dca-832a-ab31aa9991ce-host\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-conf-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.188908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-multus-daemon-config\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-node-log\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-env-overrides\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.188976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-k8s-cni-cncf-io\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-multus\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.189161 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-sys-fs\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-netns\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-cni-binary-copy\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-etc-kubernetes\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bmv\" (UniqueName: \"kubernetes.io/projected/0667514c-9805-40c3-8012-d36886ba93d2-kube-api-access-p9bmv\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-slash\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlf7\" (UniqueName: \"kubernetes.io/projected/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-kube-api-access-ltlf7\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.189523 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-os-release\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ccn\" (UniqueName: \"kubernetes.io/projected/5d0a302e-ec88-4382-9476-213b11b253f4-kube-api-access-56ccn\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-tuned\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189641 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-bin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-dbus\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.189949 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkw8j\" (UniqueName: \"kubernetes.io/projected/01e170ce-67ac-4dca-832a-ab31aa9991ce-kube-api-access-jkw8j\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.189945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-cnibin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-multus-certs\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-log-socket\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-netd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-host\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5cb\" (UniqueName: \"kubernetes.io/projected/52a87494-68cd-4205-96cf-57bebf61dcdf-kube-api-access-5r5cb\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.190216 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-modprobe-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-var-lib-kubelet\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-os-release\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-systemd\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-tmp-dir\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.190556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-kubernetes\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.190869 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-system-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190869 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbbd\" (UniqueName: \"kubernetes.io/projected/d891e983-ecf4-43b5-a414-942ab4c98826-kube-api-access-4cbbd\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.190869 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:04.191005 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-iptables-alerter-script\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.191005 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.190964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlk6\" (UniqueName: \"kubernetes.io/projected/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-kube-api-access-znlk6\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.191095 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-etc-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.191095 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysconfig\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.191184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-conf\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.191184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-run\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-host-slash\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-script-lib\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-lib-modules\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-kubelet\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-systemd-units\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-sys\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-tmp\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-agent-certs\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-config\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.191871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-bin\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.192285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.192133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-hosts-file\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.193169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.192232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01e170ce-67ac-4dca-832a-ab31aa9991ce-serviceca\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.193169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.192268 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctv5\" (UniqueName: \"kubernetes.io/projected/1cb80383-d033-4d43-b52b-b5cd250b9b76-kube-api-access-qctv5\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.195728 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.195971 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196014 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196203 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196230 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196293 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196402 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196506 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196551 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w4mlq\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 15:33:04.197046 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.196943 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnszd\"" Jul 28 15:33:04.197685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 15:33:04.197685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.197685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197432 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pfs8z\"" Jul 28 15:33:04.197685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 15:33:04.197685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197678 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fpxzh\"" Jul 28 15:33:04.197918 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197787 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 15:33:04.197970 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.197956 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 15:33:04.198095 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.198083 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sjfl2\"" Jul 28 15:33:04.206240 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.206217 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 15:33:04.233255 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.233215 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rrzm" Jul 28 15:33:04.252867 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.252842 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rrzm" Jul 28 15:33:04.275959 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.275930 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 15:33:04.293082 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-systemd\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-tmp-dir\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-kubernetes\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-system-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbbd\" (UniqueName: \"kubernetes.io/projected/d891e983-ecf4-43b5-a414-942ab4c98826-kube-api-access-4cbbd\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-systemd\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293246 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-kubernetes\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-iptables-alerter-script\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlk6\" (UniqueName: \"kubernetes.io/projected/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-kube-api-access-znlk6\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-etc-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysconfig\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-conf\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-run\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-host-slash\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.293502 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-script-lib\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-lib-modules\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-kubelet\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-system-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-tmp-dir\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-systemd-units\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-sys\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-tmp\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-agent-certs\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.293776 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-run\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-config\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-systemd-units\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-kubelet\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-bin\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-hosts-file\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01e170ce-67ac-4dca-832a-ab31aa9991ce-serviceca\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-etc-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctv5\" (UniqueName: \"kubernetes.io/projected/1cb80383-d033-4d43-b52b-b5cd250b9b76-kube-api-access-qctv5\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysconfig\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-socket-dir-parent\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.294143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-hostroot\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-kubelet-config\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-systemd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-cnibin\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-conf\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294254 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-host-slash\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.293963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-sys\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-bin\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-sysctl-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-hosts-file\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-socket-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.294634 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-lib-modules\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.295108 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01e170ce-67ac-4dca-832a-ab31aa9991ce-serviceca\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295155 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-socket-dir-parent\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-config\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-hostroot\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-cnibin\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-systemd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-ovnkube-script-lib\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.294271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-socket-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-kubelet-config\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.295339 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-device-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-kubelet\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-var-lib-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52a87494-68cd-4205-96cf-57bebf61dcdf-ovn-node-metrics-cert\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-kubelet\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rb44\" (UniqueName: \"kubernetes.io/projected/30fe3678-fed3-49a2-8b35-1eed25e9c327-kube-api-access-4rb44\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-device-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.296425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-netns\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-var-lib-openvswitch\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-konnectivity-ca\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.295654 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:04.795632112 +0000 UTC m=+2.996479581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-registration-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-ovn\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01e170ce-67ac-4dca-832a-ab31aa9991ce-host\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-conf-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-multus-daemon-config\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-iptables-alerter-script\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-node-log\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-env-overrides\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-k8s-cni-cncf-io\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295848 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-node-log\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-multus\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.297362 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-sys-fs\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-netns\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.295996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-netns\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-cni-binary-copy\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-konnectivity-ca\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-etc-kubernetes\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-run-ovn\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bmv\" (UniqueName: \"kubernetes.io/projected/0667514c-9805-40c3-8012-d36886ba93d2-kube-api-access-p9bmv\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-slash\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-netns\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlf7\" (UniqueName: \"kubernetes.io/projected/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-kube-api-access-ltlf7\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-os-release\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298160 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ccn\" (UniqueName: \"kubernetes.io/projected/5d0a302e-ec88-4382-9476-213b11b253f4-kube-api-access-56ccn\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-tuned\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-bin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296478 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-conf-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52a87494-68cd-4205-96cf-57bebf61dcdf-env-overrides\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-k8s-cni-cncf-io\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-dbus\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-registration-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkw8j\" (UniqueName: \"kubernetes.io/projected/01e170ce-67ac-4dca-832a-ab31aa9991ce-kube-api-access-jkw8j\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-multus\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-cnibin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-dbus\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-multus-certs\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-cnibin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-run-multus-certs\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296700 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-log-socket\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.298809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-netd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-host\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5cb\" (UniqueName: \"kubernetes.io/projected/52a87494-68cd-4205-96cf-57bebf61dcdf-kube-api-access-5r5cb\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-modprobe-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-slash\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-var-lib-kubelet\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0667514c-9805-40c3-8012-d36886ba93d2-sys-fs\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-os-release\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.296998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-etc-kubernetes\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297088 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-os-release\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-host\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-log-socket\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-cni-netd\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01e170ce-67ac-4dca-832a-ab31aa9991ce-host\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a87494-68cd-4205-96cf-57bebf61dcdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-cni-binary-copy\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.299532 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d891e983-ecf4-43b5-a414-942ab4c98826-multus-daemon-config\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.297940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d0a302e-ec88-4382-9476-213b11b253f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.298007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-modprobe-d\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.298054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cb80383-d033-4d43-b52b-b5cd250b9b76-var-lib-kubelet\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.298101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-multus-cni-dir\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.298126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d0a302e-ec88-4382-9476-213b11b253f4-os-release\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.300442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.298172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d891e983-ecf4-43b5-a414-942ab4c98826-host-var-lib-cni-bin\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.300784 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.300704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52a87494-68cd-4205-96cf-57bebf61dcdf-ovn-node-metrics-cert\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.300922 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.300904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-etc-tuned\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.301288 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.301269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abd8e8-0dd4-4277-82cd-2f94b82b6f33-agent-certs\") pod \"konnectivity-agent-fr47v\" (UID: \"99abd8e8-0dd4-4277-82cd-2f94b82b6f33\") " pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.303576 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.303541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cb80383-d033-4d43-b52b-b5cd250b9b76-tmp\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.314915 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.314892 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:04.314915 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.314918 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:04.315043 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.314929 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:04.315043 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.315002 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:04.814982016 +0000 UTC m=+3.015829489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:04.327857 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.327828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlf7\" (UniqueName: \"kubernetes.io/projected/02fe1d62-7f94-4b61-ab13-e8bdeaea5870-kube-api-access-ltlf7\") pod \"node-resolver-mb5q9\" (UID: \"02fe1d62-7f94-4b61-ab13-e8bdeaea5870\") " pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.328006 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.327855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlk6\" (UniqueName: \"kubernetes.io/projected/5c30af1a-ffb8-4255-8c8d-e4b35eecccd0-kube-api-access-znlk6\") pod \"iptables-alerter-5c5fk\" (UID: \"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0\") " pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.332898 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.332865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bmv\" (UniqueName: \"kubernetes.io/projected/0667514c-9805-40c3-8012-d36886ba93d2-kube-api-access-p9bmv\") pod \"aws-ebs-csi-driver-node-9dmsn\" (UID: \"0667514c-9805-40c3-8012-d36886ba93d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.335643 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.335530 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5cb\" (UniqueName: \"kubernetes.io/projected/52a87494-68cd-4205-96cf-57bebf61dcdf-kube-api-access-5r5cb\") pod \"ovnkube-node-kgsxf\" (UID: \"52a87494-68cd-4205-96cf-57bebf61dcdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.336197 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.336152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ccn\" (UniqueName: \"kubernetes.io/projected/5d0a302e-ec88-4382-9476-213b11b253f4-kube-api-access-56ccn\") pod \"multus-additional-cni-plugins-nxmtr\" (UID: \"5d0a302e-ec88-4382-9476-213b11b253f4\") " pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.336298 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.336204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbbd\" (UniqueName: \"kubernetes.io/projected/d891e983-ecf4-43b5-a414-942ab4c98826-kube-api-access-4cbbd\") pod \"multus-6vpqn\" (UID: \"d891e983-ecf4-43b5-a414-942ab4c98826\") " pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.336893 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.336846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctv5\" (UniqueName: \"kubernetes.io/projected/1cb80383-d033-4d43-b52b-b5cd250b9b76-kube-api-access-qctv5\") pod \"tuned-crt5d\" (UID: \"1cb80383-d033-4d43-b52b-b5cd250b9b76\") " pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.338005 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.337985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkw8j\" (UniqueName: \"kubernetes.io/projected/01e170ce-67ac-4dca-832a-ab31aa9991ce-kube-api-access-jkw8j\") pod \"node-ca-sj6zz\" (UID: \"01e170ce-67ac-4dca-832a-ab31aa9991ce\") " pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.397984 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.397895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.397984 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.397953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rb44\" (UniqueName: \"kubernetes.io/projected/30fe3678-fed3-49a2-8b35-1eed25e9c327-kube-api-access-4rb44\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.398339 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.398318 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:04.398434 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.398402 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:04.898368541 +0000 UTC m=+3.099216013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:04.426855 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.426807 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rb44\" (UniqueName: \"kubernetes.io/projected/30fe3678-fed3-49a2-8b35-1eed25e9c327-kube-api-access-4rb44\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.475993 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.475959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" Jul 28 15:33:04.484717 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.484693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:04.497784 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.497749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sj6zz" Jul 28 15:33:04.504755 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.504732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crt5d" Jul 28 15:33:04.512562 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.512533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vpqn" Jul 28 15:33:04.528556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.528529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" Jul 28 15:33:04.537423 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.537386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5c5fk" Jul 28 15:33:04.545678 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.545649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:04.552098 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.552078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mb5q9" Jul 28 15:33:04.802131 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.802097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:04.802302 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.802239 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:04.802345 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.802317 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:05.802296956 +0000 UTC m=+4.003144452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:04.902560 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.902529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:04.902767 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:04.902590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:04.902767 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902734 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:04.902767 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902740 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:04.902767 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902761 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:04.902767 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902770 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:04.903027 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:05.902782129 +0000 UTC m=+4.103629598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:04.903027 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:04.902830 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:05.902806678 +0000 UTC m=+4.103654164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:05.244114 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.244029 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:05.255012 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.254974 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:04 +0000 UTC" deadline="2028-01-03 02:07:03.411731228 +0000 UTC" Jul 28 15:33:05.255012 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.255009 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12562h33m58.156727463s" Jul 28 15:33:05.343767 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.343741 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c30af1a_ffb8_4255_8c8d_e4b35eecccd0.slice/crio-b4117389de9be6a45cfea299a52307e959f62a4b67dcb7c215a493a386a33451 WatchSource:0}: Error finding container b4117389de9be6a45cfea299a52307e959f62a4b67dcb7c215a493a386a33451: Status 404 returned error can't find the container with id b4117389de9be6a45cfea299a52307e959f62a4b67dcb7c215a493a386a33451 Jul 28 15:33:05.345638 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.345510 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0667514c_9805_40c3_8012_d36886ba93d2.slice/crio-fe074220ded21ad3b6edf96221109ef63ef8351aa396c7c9bc3e6f712c0d23ad WatchSource:0}: Error finding container fe074220ded21ad3b6edf96221109ef63ef8351aa396c7c9bc3e6f712c0d23ad: Status 404 returned error can't find the container with id fe074220ded21ad3b6edf96221109ef63ef8351aa396c7c9bc3e6f712c0d23ad Jul 28 15:33:05.346179 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.345980 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e170ce_67ac_4dca_832a_ab31aa9991ce.slice/crio-00fbb74c51f2cccc5d71d0c21b388826b37161e38020cf58a642e61afff85e6a WatchSource:0}: Error finding container 00fbb74c51f2cccc5d71d0c21b388826b37161e38020cf58a642e61afff85e6a: Status 404 returned error can't find the container with id 00fbb74c51f2cccc5d71d0c21b388826b37161e38020cf58a642e61afff85e6a Jul 28 15:33:05.346783 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.346762 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb80383_d033_4d43_b52b_b5cd250b9b76.slice/crio-49e0d876b3a0f61f73ba0bb8f55208bfa41bd0c6c946e8129b34e1182d811c80 WatchSource:0}: Error finding container 49e0d876b3a0f61f73ba0bb8f55208bfa41bd0c6c946e8129b34e1182d811c80: Status 404 returned error can't find the container with id 49e0d876b3a0f61f73ba0bb8f55208bfa41bd0c6c946e8129b34e1182d811c80 Jul 28 15:33:05.349172 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.348057 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd891e983_ecf4_43b5_a414_942ab4c98826.slice/crio-79664b3aa0b98cc29b929639b7c091088e27af08c244615b5c5076f1104211bc WatchSource:0}: Error finding container 79664b3aa0b98cc29b929639b7c091088e27af08c244615b5c5076f1104211bc: Status 404 returned error can't find the container with id 79664b3aa0b98cc29b929639b7c091088e27af08c244615b5c5076f1104211bc Jul 28 15:33:05.349172 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.348779 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a87494_68cd_4205_96cf_57bebf61dcdf.slice/crio-fd2ae3434142bd9e3d3df59affc18700f3fef12ee1320ea3598ca3889d642d6f WatchSource:0}: Error finding container fd2ae3434142bd9e3d3df59affc18700f3fef12ee1320ea3598ca3889d642d6f: Status 404 returned error can't find the container with id fd2ae3434142bd9e3d3df59affc18700f3fef12ee1320ea3598ca3889d642d6f Jul 28 15:33:05.353306 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.352888 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99abd8e8_0dd4_4277_82cd_2f94b82b6f33.slice/crio-8ad094ef54bb94e733ffb16dadbee14c057edc3aa813ab0be1223818bd8aa555 WatchSource:0}: Error finding container 8ad094ef54bb94e733ffb16dadbee14c057edc3aa813ab0be1223818bd8aa555: Status 404 returned error can't find the container with id 8ad094ef54bb94e733ffb16dadbee14c057edc3aa813ab0be1223818bd8aa555 Jul 28 15:33:05.355980 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.355947 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0a302e_ec88_4382_9476_213b11b253f4.slice/crio-394631733ab352c1b610d4fb851a3275ff4613fda9d119c29e214c8c401b0ff2 WatchSource:0}: Error finding container 394631733ab352c1b610d4fb851a3275ff4613fda9d119c29e214c8c401b0ff2: Status 404 returned error can't find the container with id 394631733ab352c1b610d4fb851a3275ff4613fda9d119c29e214c8c401b0ff2 Jul 28 15:33:05.356493 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:05.356469 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fe1d62_7f94_4b61_ab13_e8bdeaea5870.slice/crio-47b2d936f080d4743f3ddb62f15e8d0892f4f5db3b28561c389df3250d2af7a6 WatchSource:0}: Error finding container 47b2d936f080d4743f3ddb62f15e8d0892f4f5db3b28561c389df3250d2af7a6: Status 404 returned error can't find the container with id 47b2d936f080d4743f3ddb62f15e8d0892f4f5db3b28561c389df3250d2af7a6 Jul 28 15:33:05.376190 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.376165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerStarted","Data":"394631733ab352c1b610d4fb851a3275ff4613fda9d119c29e214c8c401b0ff2"} Jul 28 15:33:05.377375 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.377353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"fd2ae3434142bd9e3d3df59affc18700f3fef12ee1320ea3598ca3889d642d6f"} Jul 28 15:33:05.378480 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.378452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sj6zz" event={"ID":"01e170ce-67ac-4dca-832a-ab31aa9991ce","Type":"ContainerStarted","Data":"00fbb74c51f2cccc5d71d0c21b388826b37161e38020cf58a642e61afff85e6a"} Jul 28 15:33:05.379915 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.379891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5c5fk" event={"ID":"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0","Type":"ContainerStarted","Data":"b4117389de9be6a45cfea299a52307e959f62a4b67dcb7c215a493a386a33451"} Jul 28 15:33:05.383170 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.382106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fr47v" event={"ID":"99abd8e8-0dd4-4277-82cd-2f94b82b6f33","Type":"ContainerStarted","Data":"8ad094ef54bb94e733ffb16dadbee14c057edc3aa813ab0be1223818bd8aa555"} Jul 28 15:33:05.386764 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.386690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vpqn" event={"ID":"d891e983-ecf4-43b5-a414-942ab4c98826","Type":"ContainerStarted","Data":"79664b3aa0b98cc29b929639b7c091088e27af08c244615b5c5076f1104211bc"} Jul 28 15:33:05.388611 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.388589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crt5d" event={"ID":"1cb80383-d033-4d43-b52b-b5cd250b9b76","Type":"ContainerStarted","Data":"49e0d876b3a0f61f73ba0bb8f55208bfa41bd0c6c946e8129b34e1182d811c80"} Jul 28 15:33:05.394199 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.394172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" event={"ID":"0667514c-9805-40c3-8012-d36886ba93d2","Type":"ContainerStarted","Data":"fe074220ded21ad3b6edf96221109ef63ef8351aa396c7c9bc3e6f712c0d23ad"} Jul 28 15:33:05.395113 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.395092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mb5q9" event={"ID":"02fe1d62-7f94-4b61-ab13-e8bdeaea5870","Type":"ContainerStarted","Data":"47b2d936f080d4743f3ddb62f15e8d0892f4f5db3b28561c389df3250d2af7a6"} Jul 28 15:33:05.816200 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.815952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:05.816370 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.816220 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:05.816370 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.816285 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:07.816266236 +0000 UTC m=+6.017113705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:05.917420 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.917369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:05.917420 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:05.917453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:05.917724 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.917600 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:05.917724 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.917660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:07.917641237 +0000 UTC m=+6.118488711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:05.918072 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.918050 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:05.918150 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.918079 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:05.918150 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.918092 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:05.918150 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:05.918135 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:07.918121462 +0000 UTC m=+6.118968934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:06.255576 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.255474 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:04 +0000 UTC" deadline="2028-02-15 15:41:42.184383528 +0000 UTC" Jul 28 15:33:06.255576 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.255539 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13608h8m35.928849317s" Jul 28 15:33:06.370217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.370183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:06.370385 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:06.370315 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:06.370618 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.370528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:06.370706 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:06.370631 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:06.370706 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.370660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:06.370807 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:06.370733 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:06.413749 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.413711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" event={"ID":"82fc34b6950a9b1738a19b765063a610","Type":"ContainerStarted","Data":"28410c07d424de032f5a189aa2b1417621e3d6bc590508b91c454ac99f4e96d0"} Jul 28 15:33:06.417305 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.417003 2575 generic.go:356] "Generic (PLEG): container finished" podID="d0ec366b83bd37199ae0b62f4af0e339" containerID="6ba21d1b908fa26d4e63997ddc52a1f10d266482984c5512f56d7551fe555698" exitCode=0 Jul 28 15:33:06.417305 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.417049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" event={"ID":"d0ec366b83bd37199ae0b62f4af0e339","Type":"ContainerDied","Data":"6ba21d1b908fa26d4e63997ddc52a1f10d266482984c5512f56d7551fe555698"} Jul 28 15:33:06.465977 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:06.465897 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-8.ec2.internal" podStartSLOduration=3.4658745890000002 podStartE2EDuration="3.465874589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:06.439542641 +0000 UTC m=+4.640390133" watchObservedRunningTime="2026-07-28 15:33:06.465874589 +0000 UTC m=+4.666722082" Jul 28 15:33:07.433036 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:07.432956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" event={"ID":"d0ec366b83bd37199ae0b62f4af0e339","Type":"ContainerStarted","Data":"66d243fba076669a5b5b0f7813d0779462a6856d8027d68f5a63c3dc39b637d3"} Jul 28 15:33:07.835202 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:07.834630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:07.835202 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.834797 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:07.835202 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.834863 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.834845467 +0000 UTC m=+10.035692942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:07.935568 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:07.935531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:07.935751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:07.935607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:07.935751 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935705 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:07.935875 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935759 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:07.935875 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935770 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:07.935875 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935783 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:07.935875 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935832 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.935813212 +0000 UTC m=+10.136660688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:07.935875 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:07.935852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.935841702 +0000 UTC m=+10.136689177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:08.367660 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:08.367631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:08.367830 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:08.367762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:08.367957 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:08.367937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:08.368168 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:08.368060 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:08.368168 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:08.368142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:08.368317 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:08.368229 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:10.367935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:10.368084 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:10.368209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:10.368334 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:10.368371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:10.368484 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:10.368466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:11.871829 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:11.871789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:11.872423 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.871970 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:11.872423 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.872054 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:19.872032379 +0000 UTC m=+18.072879862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:11.972830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:11.972917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973007 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973079 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973102 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973137 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973084 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:19.973063655 +0000 UTC m=+18.173911124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:11.973238 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:11.973200 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:19.973183442 +0000 UTC m=+18.174030911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:12.368121 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:12.368080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:12.368291 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:12.368163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:12.368291 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:12.368192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:12.368291 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:12.368227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:12.368291 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:12.368254 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:12.368547 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:12.368308 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:14.371092 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:14.371047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:14.371593 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:14.371180 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:14.371593 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:14.371047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:14.371593 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:14.371294 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:14.371593 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:14.371047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:14.371593 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:14.371367 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:16.370955 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:16.370924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:16.370955 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:16.370957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:16.371409 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:16.370927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:16.371409 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:16.371033 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:16.371409 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:16.371102 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:16.371409 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:16.371187 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:18.367908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:18.367862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:18.367908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:18.367895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:18.368407 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:18.367863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:18.368407 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:18.367989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:18.368407 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:18.368114 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:18.368407 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:18.368214 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:19.932403 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:19.932282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:19.932939 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:19.932458 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:19.932939 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:19.932521 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:33:35.932504854 +0000 UTC m=+34.133352322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:20.032953 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:20.032901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:20.032978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033055 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033066 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033076 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033089 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033126 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.03310925 +0000 UTC m=+34.233956720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:20.033141 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.033142 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.033134552 +0000 UTC m=+34.233982026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:20.371031 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:20.370998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:20.371213 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:20.371025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:20.371213 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.371109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:20.371213 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:20.370997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:20.371213 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.371189 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:20.371468 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:20.371261 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:22.368297 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:22.368250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:22.368802 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:22.368375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:22.368802 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:22.368518 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:22.368802 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:22.368520 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:22.368802 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:22.368373 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:22.368802 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:22.368612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:24.368262 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.368060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:24.369114 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.368068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:24.369114 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:24.368374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:24.369114 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.368145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:24.369114 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:24.368479 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:24.369114 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:24.368617 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:24.468566 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.468533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fr47v" event={"ID":"99abd8e8-0dd4-4277-82cd-2f94b82b6f33","Type":"ContainerStarted","Data":"fee3fb9722ddeea0fed35c525c227f367d4ce0374f65c68fbd44fa8b8aff6483"} Jul 28 15:33:24.470127 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.470101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vpqn" event={"ID":"d891e983-ecf4-43b5-a414-942ab4c98826","Type":"ContainerStarted","Data":"51cabadca9d7a281d0a88d9e5984099cb69f7fc93893cd195c9f8d6b06e0ada8"} Jul 28 15:33:24.471600 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.471576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crt5d" event={"ID":"1cb80383-d033-4d43-b52b-b5cd250b9b76","Type":"ContainerStarted","Data":"2c5d879eeea72183fd9555d68975fab6b580ac9dee2601c881c06a10ca898815"} Jul 28 15:33:24.473107 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.473089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" event={"ID":"0667514c-9805-40c3-8012-d36886ba93d2","Type":"ContainerStarted","Data":"919d56081cdb14af9a0783c3309e350152069cc477aacde0dd797e00adf50fb8"} Jul 28 15:33:24.474412 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.474371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mb5q9" event={"ID":"02fe1d62-7f94-4b61-ab13-e8bdeaea5870","Type":"ContainerStarted","Data":"d067e0b74bf3921547b7c014893b200c1b2ae07c8d06e4ebee7c426ac59e4304"} Jul 28 15:33:24.476009 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.475980 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="4ae92a01fe7f34ab1566c04fce3634cb5fff581dc7cdd29d347c5431d7139560" exitCode=0 Jul 28 15:33:24.476078 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.476060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"4ae92a01fe7f34ab1566c04fce3634cb5fff581dc7cdd29d347c5431d7139560"} Jul 28 15:33:24.478898 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.478836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"fc7c57890b1ed7de9019412bb0108cb7d783a519c6fdff19bfd8b28a4646ef47"} Jul 28 15:33:24.478898 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.478895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"d586f0c7cd51ce4279f72269ff472e50fef0623d9ab85a1dfc17e0f55260422e"} Jul 28 15:33:24.479015 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.478965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"9e8a867a31875a0753a3ae5927b2e1ed9fd10dffca37f558b2d165363101a571"} Jul 28 15:33:24.479083 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.479018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"ade946ff5bc3e650fd2cb734c77da6314e89b42354f48834e56878cc1b130f8f"} Jul 28 15:33:24.480206 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.480183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sj6zz" event={"ID":"01e170ce-67ac-4dca-832a-ab31aa9991ce","Type":"ContainerStarted","Data":"214531c1f42be74b2b0366eb0187454aa9a016e33bd35589369736df97b4d937"} Jul 28 15:33:24.490535 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.490498 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fr47v" podStartSLOduration=8.113060695 podStartE2EDuration="22.490486521s" totalImagesPullingTime="14.377425826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:24.490378898 +0000 UTC m=+22.691226390" watchObservedRunningTime="2026-07-28 15:33:24.490486521 +0000 UTC m=+22.691334011" Jul 28 15:33:24.490626 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.490599 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-8.ec2.internal" podStartSLOduration=21.490595368 podStartE2EDuration="21.490595368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:07.462033709 +0000 UTC m=+5.662881201" watchObservedRunningTime="2026-07-28 15:33:24.490595368 +0000 UTC m=+22.691442858" Jul 28 15:33:24.512734 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.512696 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sj6zz" podStartSLOduration=4.03900741 podStartE2EDuration="22.512681334s" totalImagesPullingTime="18.473673924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:24.512654859 +0000 UTC m=+22.713502349" watchObservedRunningTime="2026-07-28 15:33:24.512681334 +0000 UTC m=+22.713528819" Jul 28 15:33:24.588756 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.588664 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-crt5d" podStartSLOduration=4.073986891 podStartE2EDuration="22.588648459s" totalImagesPullingTime="18.514661568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:24.565045568 +0000 UTC m=+22.765893060" watchObservedRunningTime="2026-07-28 15:33:24.588648459 +0000 UTC m=+22.789495949" Jul 28 15:33:24.609086 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.609042 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6vpqn" podStartSLOduration=3.986898054 podStartE2EDuration="22.609027391s" totalImagesPullingTime="18.622129337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:24.589021016 +0000 UTC m=+22.789868508" watchObservedRunningTime="2026-07-28 15:33:24.609027391 +0000 UTC m=+22.809874883" Jul 28 15:33:24.609579 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.609552 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mb5q9" podStartSLOduration=4.120439434 podStartE2EDuration="22.609545583s" totalImagesPullingTime="18.489106149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:24.609435282 +0000 UTC m=+22.810282772" watchObservedRunningTime="2026-07-28 15:33:24.609545583 +0000 UTC m=+22.810393073" Jul 28 15:33:24.812251 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.812213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:24.895101 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:24.895073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:25.484759 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.484712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5c5fk" event={"ID":"5c30af1a-ffb8-4255-8c8d-e4b35eecccd0","Type":"ContainerStarted","Data":"aa3b46d2bbb2aa09cd1003d940965d975863be97e9ba154fd1b2419fd8c88853"} Jul 28 15:33:25.487997 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.487934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"1479225c93286149bd981ec4dc0e8eabeb4683eaef90e828bb96f3f321a0d421"} Jul 28 15:33:25.488123 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.488015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"a8f70ad3bcb391b27fa9c09472e262c307aff1cfd7726d9c350a57b7b86b8129"} Jul 28 15:33:25.501904 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.501746 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5c5fk" podStartSLOduration=4.984831862 podStartE2EDuration="23.501724942s" totalImagesPullingTime="18.51689308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:25.501618109 +0000 UTC m=+23.702465601" watchObservedRunningTime="2026-07-28 15:33:25.501724942 +0000 UTC m=+23.702572435" Jul 28 15:33:25.602760 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.602730 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 15:33:25.634788 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.634751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:25.703671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:25.703586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:26.246672 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.246549 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:33:25.602753815Z","UUID":"41fae1cb-0405-4a51-8e0e-37797cb440db","Handler":null,"Name":"","Endpoint":""} Jul 28 15:33:26.248221 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.248195 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 15:33:26.248447 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.248239 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 15:33:26.368109 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.368082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:26.368109 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.368099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:26.368349 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:26.368207 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:26.368349 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.368243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:26.368459 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:26.368349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:26.368514 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:26.368475 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:26.492626 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.492509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" event={"ID":"0667514c-9805-40c3-8012-d36886ba93d2","Type":"ContainerStarted","Data":"eb6fc44c918c4753959b56112c9c7bd2ffead058a16e13d5c661a32ad6de462c"} Jul 28 15:33:26.584270 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.584244 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:26.584917 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:26.584898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:27.496671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.496630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" event={"ID":"0667514c-9805-40c3-8012-d36886ba93d2","Type":"ContainerStarted","Data":"166a1120529a494756b8d22b21a37a59e8a4bbef7b8b5506c27c06573c0f4885"} Jul 28 15:33:27.500137 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.500109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"80ee71edd2e04fd2a33f00ce0128ca4d2218195d4dcda7e1f2f4ed1c9f7cf36b"} Jul 28 15:33:27.500292 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.500270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:27.500778 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.500760 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fr47v" Jul 28 15:33:27.552697 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.552637 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dmsn" podStartSLOduration=4.648959475 podStartE2EDuration="25.552615991s" totalImagesPullingTime="20.903656516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:27.52809748 +0000 UTC m=+25.728944972" watchObservedRunningTime="2026-07-28 15:33:27.552615991 +0000 UTC m=+25.753463482" Jul 28 15:33:27.610235 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:27.610202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:28.147209 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.147160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:28.254442 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.254384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:28.357196 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.357161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" limit=1024 Jul 28 15:33:28.367650 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.367614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:28.367828 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.367614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:28.367828 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:28.367757 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:28.367946 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:28.367838 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:28.367946 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:28.367614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:28.367946 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:28.367925 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:29.505869 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.505691 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="3f0bbc25e05d57054ec0e03d7b4f74005263c7738a1495e31eae8f7f1c165763" exitCode=0 Jul 28 15:33:29.506746 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.505749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"3f0bbc25e05d57054ec0e03d7b4f74005263c7738a1495e31eae8f7f1c165763"} Jul 28 15:33:29.509781 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.509752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" event={"ID":"52a87494-68cd-4205-96cf-57bebf61dcdf","Type":"ContainerStarted","Data":"6e5ef2982e6ec916c1944bc72fde8fd55249561ab6ac633803e9e73ad04a2fe2"} Jul 28 15:33:29.510053 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.510028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:29.510173 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.510100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:29.525700 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.525677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:29.573042 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:29.572981 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" podStartSLOduration=8.911967473 podStartE2EDuration="27.572967026s" totalImagesPullingTime="18.660999553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:29.572618395 +0000 UTC m=+27.773465887" watchObservedRunningTime="2026-07-28 15:33:29.572967026 +0000 UTC m=+27.773814517" Jul 28 15:33:30.368095 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:30.368062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:30.368302 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:30.368063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:30.368302 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:30.368176 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:30.368384 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:30.368063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:30.368384 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:30.368333 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:30.368485 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:30.368453 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:30.512466 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:30.512373 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:30.528332 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:30.528130 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:33:31.103069 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.102982 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g965z"] Jul 28 15:33:31.103264 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.103113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:31.103264 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:31.103249 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:31.103571 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.103544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgwzd"] Jul 28 15:33:31.103655 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.103623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:31.103710 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:31.103695 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:31.105091 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.105057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7j2v2"] Jul 28 15:33:31.105183 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.105137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:31.105234 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:31.105213 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:31.516505 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.516474 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="3e2d70f7a2e476f9be88f72673bedfdacbd4d931708178468f184335317efe6c" exitCode=0 Jul 28 15:33:31.516865 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:31.516557 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"3e2d70f7a2e476f9be88f72673bedfdacbd4d931708178468f184335317efe6c"} Jul 28 15:33:32.368772 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:32.368742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:32.368951 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:32.368839 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:33.367556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:33.367526 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:33.367941 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:33.367631 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:33.367941 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:33.367688 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:33.367941 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:33.367782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:33.523374 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:33.523339 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="97676cf6971d3772bfeffd539f26434b97b68749acb8584ecfa64ce37b19c1e2" exitCode=0 Jul 28 15:33:33.523556 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:33.523429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"97676cf6971d3772bfeffd539f26434b97b68749acb8584ecfa64ce37b19c1e2"} Jul 28 15:33:34.367773 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:34.367530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:34.368230 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:34.367871 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:35.367488 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:35.367454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:35.367672 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:35.367454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:35.367672 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:35.367573 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j2v2" podUID="505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27" Jul 28 15:33:35.367778 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:35.367682 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgwzd" podUID="43a7b6a7-b8df-412b-82fa-ba25e8ea512f" Jul 28 15:33:35.946581 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:35.946486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:35.947063 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:35.946659 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:35.947063 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:35.946728 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret podName:43a7b6a7-b8df-412b-82fa-ba25e8ea512f nodeName:}" failed. No retries permitted until 2026-07-28 15:34:07.946710524 +0000 UTC m=+66.147558010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret") pod "global-pull-secret-syncer-wgwzd" (UID: "43a7b6a7-b8df-412b-82fa-ba25e8ea512f") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:36.047090 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.047054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:36.047290 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.047210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:36.047290 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047246 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:36.047290 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047276 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:36.047290 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047290 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rlmfj for pod openshift-network-diagnostics/network-check-target-7j2v2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:36.047550 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047323 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:36.047550 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047357 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj podName:505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.047337012 +0000 UTC m=+66.248184484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmfj" (UniqueName: "kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj") pod "network-check-target-7j2v2" (UID: "505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:36.047550 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.047378 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs podName:30fe3678-fed3-49a2-8b35-1eed25e9c327 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.047367664 +0000 UTC m=+66.248215138 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs") pod "network-metrics-daemon-g965z" (UID: "30fe3678-fed3-49a2-8b35-1eed25e9c327") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:36.367700 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.367670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:36.367898 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.367812 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g965z" podUID="30fe3678-fed3-49a2-8b35-1eed25e9c327" Jul 28 15:33:36.627564 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.627483 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-8.ec2.internal" event="NodeReady" Jul 28 15:33:36.627707 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.627661 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 15:33:36.718831 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.718796 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:33:36.721877 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.721853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.728476 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.728455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 15:33:36.734872 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.734850 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 15:33:36.740849 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.740824 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 15:33:36.740960 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.740899 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 15:33:36.766603 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.766566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7nwrd\"" Jul 28 15:33:36.776989 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.776957 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kzq4z"] Jul 28 15:33:36.779039 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.779009 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5br4j"] Jul 28 15:33:36.779188 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.779131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:36.780894 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.780872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:36.810324 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.810285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 15:33:36.810501 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.810288 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 15:33:36.810501 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.810385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ks94b\"" Jul 28 15:33:36.810751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.810733 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 15:33:36.810854 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.810766 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2rz82\"" Jul 28 15:33:36.811199 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.811182 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 15:33:36.819050 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.819028 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 15:33:36.823965 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.823940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5br4j"] Jul 28 15:33:36.828887 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.828860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kzq4z"] Jul 28 15:33:36.829735 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.829714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:33:36.853401 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2c2\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.853751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.853680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.954718 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.954718 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-tmp-dir\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-config-volume\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.954957 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.954977 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4f76c666-5pk7c: secret "image-registry-tls" not found Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.954962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f00501-a051-4585-9d34-0e25eac0aa3f-cert\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:36.955040 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls podName:cc032a8d-a6ff-472e-9dba-d7f2451198ce nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.455017389 +0000 UTC m=+35.655864858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls") pod "image-registry-6b4f76c666-5pk7c" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce") : secret "image-registry-tls" not found Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzf5\" (UniqueName: \"kubernetes.io/projected/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-kube-api-access-svzf5\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v99\" (UniqueName: \"kubernetes.io/projected/37f00501-a051-4585-9d34-0e25eac0aa3f-kube-api-access-b5v99\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955230 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2c2\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.955783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:36.955783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.956016 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.955881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.960106 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.960083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.960232 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.960175 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.972325 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.972264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:36.975162 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:36.975139 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2c2\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:37.056436 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzf5\" (UniqueName: \"kubernetes.io/projected/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-kube-api-access-svzf5\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.056436 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v99\" (UniqueName: \"kubernetes.io/projected/37f00501-a051-4585-9d34-0e25eac0aa3f-kube-api-access-b5v99\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:37.056671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.056671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-tmp-dir\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.056671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-config-volume\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.056671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f00501-a051-4585-9d34-0e25eac0aa3f-cert\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:37.056867 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.056694 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:33:37.056867 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.056773 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls podName:1ff3ce04-c9f0-4b53-86c6-4629f7a71927 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.556750331 +0000 UTC m=+35.757597806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls") pod "dns-default-kzq4z" (UID: "1ff3ce04-c9f0-4b53-86c6-4629f7a71927") : secret "dns-default-metrics-tls" not found Jul 28 15:33:37.056992 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.056967 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-tmp-dir\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.057341 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.057311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-config-volume\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.059368 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.059346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f00501-a051-4585-9d34-0e25eac0aa3f-cert\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:37.076268 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.076233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzf5\" (UniqueName: \"kubernetes.io/projected/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-kube-api-access-svzf5\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.077787 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.077759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v99\" (UniqueName: \"kubernetes.io/projected/37f00501-a051-4585-9d34-0e25eac0aa3f-kube-api-access-b5v99\") pod \"ingress-canary-5br4j\" (UID: \"37f00501-a051-4585-9d34-0e25eac0aa3f\") " pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:37.095671 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.095634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5br4j" Jul 28 15:33:37.367787 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.367752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:33:37.367960 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.367789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:33:37.372342 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.372319 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 15:33:37.372493 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.372345 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 15:33:37.372783 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.372762 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 15:33:37.380506 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.380337 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zk6l7"] Jul 28 15:33:37.381031 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.380831 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djz96\"" Jul 28 15:33:37.383781 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.383762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.397184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.397166 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 15:33:37.397615 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.397596 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 15:33:37.397923 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.397897 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 15:33:37.397923 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.397914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hgr8\"" Jul 28 15:33:37.398224 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.398205 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 15:33:37.411093 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.411033 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zk6l7"] Jul 28 15:33:37.461119 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.461082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:37.461305 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.461229 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:33:37.461305 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.461250 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4f76c666-5pk7c: secret "image-registry-tls" not found Jul 28 15:33:37.461449 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.461328 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls podName:cc032a8d-a6ff-472e-9dba-d7f2451198ce nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.461308378 +0000 UTC m=+36.662155848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls") pod "image-registry-6b4f76c666-5pk7c" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce") : secret "image-registry-tls" not found Jul 28 15:33:37.561714 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.561683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.561920 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.561737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:37.561920 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.561764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-data-volume\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.561920 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.561814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdxq\" (UniqueName: \"kubernetes.io/projected/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-api-access-ppdxq\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.561920 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.561868 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:33:37.561920 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.561905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-crio-socket\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.562169 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.561929 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls podName:1ff3ce04-c9f0-4b53-86c6-4629f7a71927 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.561908619 +0000 UTC m=+36.762756102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls") pod "dns-default-kzq4z" (UID: "1ff3ce04-c9f0-4b53-86c6-4629f7a71927") : secret "dns-default-metrics-tls" not found Jul 28 15:33:37.562169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.562009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.662972 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.662928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663119 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-data-volume\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663119 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdxq\" (UniqueName: \"kubernetes.io/projected/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-api-access-ppdxq\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663119 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.663095 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:33:37.663255 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.663169 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.163148264 +0000 UTC m=+36.363995737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : secret "insights-runtime-extractor-tls" not found Jul 28 15:33:37.663319 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-crio-socket\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663319 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663449 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.663434 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:33:37.663511 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-crio-socket\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.663585 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:37.663572 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:38.163535985 +0000 UTC m=+36.364383475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : configmap "kube-rbac-proxy" not found Jul 28 15:33:37.663585 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.663573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-data-volume\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.677184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.677163 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdxq\" (UniqueName: \"kubernetes.io/projected/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-api-access-ppdxq\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:37.781258 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:37.781021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5br4j"] Jul 28 15:33:37.784205 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:37.784174 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f00501_a051_4585_9d34_0e25eac0aa3f.slice/crio-fdeda6d6d149454a7664eb48280fcf9f9536f4d4cb7673c4981212230f8ed6a3 WatchSource:0}: Error finding container fdeda6d6d149454a7664eb48280fcf9f9536f4d4cb7673c4981212230f8ed6a3: Status 404 returned error can't find the container with id fdeda6d6d149454a7664eb48280fcf9f9536f4d4cb7673c4981212230f8ed6a3 Jul 28 15:33:38.097790 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.097758 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ghs9f"] Jul 28 15:33:38.100460 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.100426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.107191 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.107165 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 28 15:33:38.107317 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.107243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 28 15:33:38.107317 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.107309 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 28 15:33:38.107467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.107243 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 28 15:33:38.107624 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.107610 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r7ps5\"" Jul 28 15:33:38.120915 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.120888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ghs9f"] Jul 28 15:33:38.167719 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.167681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:38.167911 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.167737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:38.167911 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.167821 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:33:38.167911 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.167836 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:33:38.167911 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.167891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.167876366 +0000 UTC m=+37.368723835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : secret "insights-runtime-extractor-tls" not found Jul 28 15:33:38.167911 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.167906 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.167898937 +0000 UTC m=+37.368746406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : configmap "kube-rbac-proxy" not found Jul 28 15:33:38.268248 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.268211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-cabundle\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.268248 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.268248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcb2\" (UniqueName: \"kubernetes.io/projected/1bf54ef7-b717-480d-8ea0-2e4f644242e0-kube-api-access-cwcb2\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.268490 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.268384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-key\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.368323 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.368243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:33:38.369109 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.369077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-key\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.370084 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.369201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-cabundle\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.370084 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.369230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcb2\" (UniqueName: \"kubernetes.io/projected/1bf54ef7-b717-480d-8ea0-2e4f644242e0-kube-api-access-cwcb2\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.370084 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.370073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-cabundle\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.371229 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.371207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hctgx\"" Jul 28 15:33:38.371742 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.371466 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 15:33:38.373507 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.373487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bf54ef7-b717-480d-8ea0-2e4f644242e0-signing-key\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.379297 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.379269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcb2\" (UniqueName: \"kubernetes.io/projected/1bf54ef7-b717-480d-8ea0-2e4f644242e0-kube-api-access-cwcb2\") pod \"service-ca-658d7b7c78-ghs9f\" (UID: \"1bf54ef7-b717-480d-8ea0-2e4f644242e0\") " pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.410088 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.410055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" Jul 28 15:33:38.472479 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.471835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:38.472479 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.471987 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:33:38.472479 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.472004 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4f76c666-5pk7c: secret "image-registry-tls" not found Jul 28 15:33:38.472479 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.472072 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls podName:cc032a8d-a6ff-472e-9dba-d7f2451198ce nodeName:}" failed. No retries permitted until 2026-07-28 15:33:40.472050605 +0000 UTC m=+38.672898077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls") pod "image-registry-6b4f76c666-5pk7c" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce") : secret "image-registry-tls" not found Jul 28 15:33:38.538760 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.538730 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="0ba6931ed160490beb5fdb7a0bcad0d4d0f0ce261ed7000ee01f7ed7a611f197" exitCode=0 Jul 28 15:33:38.538946 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.538835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"0ba6931ed160490beb5fdb7a0bcad0d4d0f0ce261ed7000ee01f7ed7a611f197"} Jul 28 15:33:38.540287 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.540258 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5br4j" event={"ID":"37f00501-a051-4585-9d34-0e25eac0aa3f","Type":"ContainerStarted","Data":"fdeda6d6d149454a7664eb48280fcf9f9536f4d4cb7673c4981212230f8ed6a3"} Jul 28 15:33:38.574193 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.573845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:38.574193 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.573956 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:33:38.574193 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:38.574018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls podName:1ff3ce04-c9f0-4b53-86c6-4629f7a71927 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:40.574001338 +0000 UTC m=+38.774848808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls") pod "dns-default-kzq4z" (UID: "1ff3ce04-c9f0-4b53-86c6-4629f7a71927") : secret "dns-default-metrics-tls" not found Jul 28 15:33:38.578518 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:38.578488 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf54ef7_b717_480d_8ea0_2e4f644242e0.slice/crio-a505a4f7236d8d428c38fc4ba465c3c5f9b1204d63ebbe5e3c4d22803f7dbb82 WatchSource:0}: Error finding container a505a4f7236d8d428c38fc4ba465c3c5f9b1204d63ebbe5e3c4d22803f7dbb82: Status 404 returned error can't find the container with id a505a4f7236d8d428c38fc4ba465c3c5f9b1204d63ebbe5e3c4d22803f7dbb82 Jul 28 15:33:38.579079 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:38.579014 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ghs9f"] Jul 28 15:33:39.179694 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.179655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:39.180164 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:39.179845 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:33:39.180164 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.179854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:39.180164 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:39.179925 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:41.179902977 +0000 UTC m=+39.380750448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : secret "insights-runtime-extractor-tls" not found Jul 28 15:33:39.180164 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:39.179953 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:33:39.180164 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:39.180004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:41.179988725 +0000 UTC m=+39.380836198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : configmap "kube-rbac-proxy" not found Jul 28 15:33:39.544725 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.544599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" event={"ID":"1bf54ef7-b717-480d-8ea0-2e4f644242e0","Type":"ContainerStarted","Data":"a505a4f7236d8d428c38fc4ba465c3c5f9b1204d63ebbe5e3c4d22803f7dbb82"} Jul 28 15:33:39.548739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.548621 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d0a302e-ec88-4382-9476-213b11b253f4" containerID="37868b7c609da031a15e489ca4c68a0023bff2326d032fc58b3eb0fd0a77302b" exitCode=0 Jul 28 15:33:39.548739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.548678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerDied","Data":"37868b7c609da031a15e489ca4c68a0023bff2326d032fc58b3eb0fd0a77302b"} Jul 28 15:33:39.617299 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:39.617266 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mb5q9_02fe1d62-7f94-4b61-ab13-e8bdeaea5870/dns-node-resolver/0.log" Jul 28 15:33:40.490958 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:40.490867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:40.491355 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:40.491054 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:33:40.491355 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:40.491078 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4f76c666-5pk7c: secret "image-registry-tls" not found Jul 28 15:33:40.491355 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:40.491149 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls podName:cc032a8d-a6ff-472e-9dba-d7f2451198ce nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.491130215 +0000 UTC m=+42.691977685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls") pod "image-registry-6b4f76c666-5pk7c" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce") : secret "image-registry-tls" not found Jul 28 15:33:40.591937 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:40.591903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:40.592110 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:40.592037 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:33:40.592110 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:40.592096 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls podName:1ff3ce04-c9f0-4b53-86c6-4629f7a71927 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:44.592082117 +0000 UTC m=+42.792929590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls") pod "dns-default-kzq4z" (UID: "1ff3ce04-c9f0-4b53-86c6-4629f7a71927") : secret "dns-default-metrics-tls" not found Jul 28 15:33:41.010624 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.010454 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sj6zz_01e170ce-67ac-4dca-832a-ab31aa9991ce/node-ca/0.log" Jul 28 15:33:41.197692 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.197656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:41.197853 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.197715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:41.197853 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:41.197785 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:33:41.197853 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:41.197845 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:33:41.197853 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:41.197852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.197834758 +0000 UTC m=+43.398682227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : configmap "kube-rbac-proxy" not found Jul 28 15:33:41.197982 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:41.197884 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.197872607 +0000 UTC m=+43.398720077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : secret "insights-runtime-extractor-tls" not found Jul 28 15:33:41.557129 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.557087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" event={"ID":"5d0a302e-ec88-4382-9476-213b11b253f4","Type":"ContainerStarted","Data":"41062e5459556a93aafe1b7991ae36cd5f49cb77b80aa6ee534b9cecf620dea0"} Jul 28 15:33:41.558580 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.558554 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5br4j" event={"ID":"37f00501-a051-4585-9d34-0e25eac0aa3f","Type":"ContainerStarted","Data":"1ea6db5c2d266c20f28f175ed4d73478bfc732c23b9225db2d1e98772e40fc4d"} Jul 28 15:33:41.559962 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.559939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" event={"ID":"1bf54ef7-b717-480d-8ea0-2e4f644242e0","Type":"ContainerStarted","Data":"b1b4928648c91ba5c31b272bb20580013256e119b5dc37b3b35648e0ff9fd1f0"} Jul 28 15:33:41.584307 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.584260 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxmtr" podStartSLOduration=10.462651702 podStartE2EDuration="39.584243189s" totalImagesPullingTime="29.121591487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:41.583155376 +0000 UTC m=+39.784002867" watchObservedRunningTime="2026-07-28 15:33:41.584243189 +0000 UTC m=+39.785090681" Jul 28 15:33:41.604769 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.604713 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5br4j" podStartSLOduration=2.694236971 podStartE2EDuration="5.60469417s" totalImagesPullingTime="2.910457199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:41.604170925 +0000 UTC m=+39.805018417" watchObservedRunningTime="2026-07-28 15:33:41.60469417 +0000 UTC m=+39.805541662" Jul 28 15:33:41.629419 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:41.629358 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-ghs9f" podStartSLOduration=1.512994306 podStartE2EDuration="3.629343978s" totalImagesPullingTime="2.116349672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:41.629092613 +0000 UTC m=+39.829940105" watchObservedRunningTime="2026-07-28 15:33:41.629343978 +0000 UTC m=+39.830191468" Jul 28 15:33:44.529264 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:44.529213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:44.529766 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:44.529383 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:33:44.529766 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:44.529426 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4f76c666-5pk7c: secret "image-registry-tls" not found Jul 28 15:33:44.529766 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:44.529484 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls podName:cc032a8d-a6ff-472e-9dba-d7f2451198ce nodeName:}" failed. No retries permitted until 2026-07-28 15:33:52.529469152 +0000 UTC m=+50.730316621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls") pod "image-registry-6b4f76c666-5pk7c" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce") : secret "image-registry-tls" not found Jul 28 15:33:44.630225 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:44.630189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:44.630435 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:44.630350 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:33:44.630435 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:44.630421 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls podName:1ff3ce04-c9f0-4b53-86c6-4629f7a71927 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:52.630406892 +0000 UTC m=+50.831254361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls") pod "dns-default-kzq4z" (UID: "1ff3ce04-c9f0-4b53-86c6-4629f7a71927") : secret "dns-default-metrics-tls" not found Jul 28 15:33:45.235454 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:45.235407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:45.235637 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:45.235585 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:33:45.235685 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:45.235640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:45.235685 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:45.235655 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:53.235636877 +0000 UTC m=+51.436484360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : secret "insights-runtime-extractor-tls" not found Jul 28 15:33:45.235790 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:45.235695 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:33:45.235790 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:33:45.235732 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm podName:f5c0f11b-9b4d-4024-ae55-e56eb381c5da nodeName:}" failed. No retries permitted until 2026-07-28 15:33:53.235723074 +0000 UTC m=+51.436570546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zk6l7" (UID: "f5c0f11b-9b4d-4024-ae55-e56eb381c5da") : configmap "kube-rbac-proxy" not found Jul 28 15:33:52.599871 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.599836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:52.603272 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.603250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"image-registry-6b4f76c666-5pk7c\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:52.633271 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.633235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:52.701032 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.700996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:52.703426 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.703351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ff3ce04-c9f0-4b53-86c6-4629f7a71927-metrics-tls\") pod \"dns-default-kzq4z\" (UID: \"1ff3ce04-c9f0-4b53-86c6-4629f7a71927\") " pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:52.778009 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.777974 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:33:52.781781 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:52.781752 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc032a8d_a6ff_472e_9dba_d7f2451198ce.slice/crio-12aa8ca8bd6a229ba8587f20ee1b71e72a166a6e5f48066df7153045c6587679 WatchSource:0}: Error finding container 12aa8ca8bd6a229ba8587f20ee1b71e72a166a6e5f48066df7153045c6587679: Status 404 returned error can't find the container with id 12aa8ca8bd6a229ba8587f20ee1b71e72a166a6e5f48066df7153045c6587679 Jul 28 15:33:52.989653 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:52.989619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:53.148189 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.148101 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kzq4z"] Jul 28 15:33:53.153469 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:53.153440 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff3ce04_c9f0_4b53_86c6_4629f7a71927.slice/crio-2d15bc450c4f81a4007d92a8ef4e17fd4492c64ed7a80cf0aaad432ff10bbaf1 WatchSource:0}: Error finding container 2d15bc450c4f81a4007d92a8ef4e17fd4492c64ed7a80cf0aaad432ff10bbaf1: Status 404 returned error can't find the container with id 2d15bc450c4f81a4007d92a8ef4e17fd4492c64ed7a80cf0aaad432ff10bbaf1 Jul 28 15:33:53.305456 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.305420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:53.305625 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.305479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:53.305942 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.305922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:53.307792 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.307767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5c0f11b-9b4d-4024-ae55-e56eb381c5da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zk6l7\" (UID: \"f5c0f11b-9b4d-4024-ae55-e56eb381c5da\") " pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:53.589356 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.589313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" event={"ID":"cc032a8d-a6ff-472e-9dba-d7f2451198ce","Type":"ContainerStarted","Data":"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519"} Jul 28 15:33:53.589578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.589372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" event={"ID":"cc032a8d-a6ff-472e-9dba-d7f2451198ce","Type":"ContainerStarted","Data":"12aa8ca8bd6a229ba8587f20ee1b71e72a166a6e5f48066df7153045c6587679"} Jul 28 15:33:53.589578 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.589440 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:33:53.590425 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.590374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzq4z" event={"ID":"1ff3ce04-c9f0-4b53-86c6-4629f7a71927","Type":"ContainerStarted","Data":"2d15bc450c4f81a4007d92a8ef4e17fd4492c64ed7a80cf0aaad432ff10bbaf1"} Jul 28 15:33:53.593530 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.593512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zk6l7" Jul 28 15:33:53.615518 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.615471 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" podStartSLOduration=26.61545396 podStartE2EDuration="26.61545396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:53.614905863 +0000 UTC m=+51.815753355" watchObservedRunningTime="2026-07-28 15:33:53.61545396 +0000 UTC m=+51.816301451" Jul 28 15:33:53.743248 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:53.743213 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zk6l7"] Jul 28 15:33:53.746758 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:33:53.746728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c0f11b_9b4d_4024_ae55_e56eb381c5da.slice/crio-72347f4bebc51e233424437f6e63ae859823417a57e35d9b402127608f6777f4 WatchSource:0}: Error finding container 72347f4bebc51e233424437f6e63ae859823417a57e35d9b402127608f6777f4: Status 404 returned error can't find the container with id 72347f4bebc51e233424437f6e63ae859823417a57e35d9b402127608f6777f4 Jul 28 15:33:54.596587 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:54.596550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zk6l7" event={"ID":"f5c0f11b-9b4d-4024-ae55-e56eb381c5da","Type":"ContainerStarted","Data":"72347f4bebc51e233424437f6e63ae859823417a57e35d9b402127608f6777f4"} Jul 28 15:33:56.604917 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:56.604871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzq4z" event={"ID":"1ff3ce04-c9f0-4b53-86c6-4629f7a71927","Type":"ContainerStarted","Data":"42994606dcc5265751145b9eda95dac2838f145143590880594a1ce27c5da5d7"} Jul 28 15:33:56.605354 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:56.604948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzq4z" event={"ID":"1ff3ce04-c9f0-4b53-86c6-4629f7a71927","Type":"ContainerStarted","Data":"962916beb306bf78d110c464760296620b0369f47fc7fecbe2920b6854ab0ed0"} Jul 28 15:33:56.605354 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:56.605061 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kzq4z" Jul 28 15:33:56.624217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:56.624163 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kzq4z" podStartSLOduration=18.245354897 podStartE2EDuration="20.624145832s" totalImagesPullingTime="2.378790935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.622831831 +0000 UTC m=+54.823679323" watchObservedRunningTime="2026-07-28 15:33:56.624145832 +0000 UTC m=+54.824993323" Jul 28 15:33:59.617412 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:33:59.617300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zk6l7" event={"ID":"f5c0f11b-9b4d-4024-ae55-e56eb381c5da","Type":"ContainerStarted","Data":"06a200000ac2729d3003f6a4762f102ede91fdc84c9f3e6c36a468fe741ab5dd"} Jul 28 15:34:01.041114 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.040737 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw"] Jul 28 15:34:01.054458 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.054427 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn"] Jul 28 15:34:01.054627 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.054595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.056981 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.056735 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 15:34:01.056981 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.056796 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-frvck\"" Jul 28 15:34:01.056981 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.056843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw"] Jul 28 15:34:01.057221 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.057077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.057363 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.057342 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 15:34:01.057666 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.057645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 28 15:34:01.058154 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.058132 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 15:34:01.059830 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.059799 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn"] Jul 28 15:34:01.060197 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.060175 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 28 15:34:01.060294 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.060263 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 28 15:34:01.060351 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.060301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 28 15:34:01.060440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.060384 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 28 15:34:01.067439 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.067415 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:34:01.141076 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.141001 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85cdd7bbb8-5886g"] Jul 28 15:34:01.145756 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.145726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.160370 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.160322 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cdd7bbb8-5886g"] Jul 28 15:34:01.175722 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.175894 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9775ec41-e21d-4485-a89e-8c489a3d26a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.175894 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vk2\" (UniqueName: \"kubernetes.io/projected/9775ec41-e21d-4485-a89e-8c489a3d26a4-kube-api-access-c7vk2\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.175894 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.175894 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.176133 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.175975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpd7\" (UniqueName: \"kubernetes.io/projected/49dffd22-1de1-4924-b13d-e0a406639b6b-kube-api-access-krpd7\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.176133 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.176008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49dffd22-1de1-4924-b13d-e0a406639b6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.176133 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.176026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.276439 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-bound-sa-token\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276439 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vng\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-kube-api-access-72vng\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276694 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-ca-trust-extracted\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276694 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-certificates\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276694 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9775ec41-e21d-4485-a89e-8c489a3d26a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.276694 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-image-registry-private-configuration\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vk2\" (UniqueName: \"kubernetes.io/projected/9775ec41-e21d-4485-a89e-8c489a3d26a4-kube-api-access-c7vk2\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpd7\" (UniqueName: \"kubernetes.io/projected/49dffd22-1de1-4924-b13d-e0a406639b6b-kube-api-access-krpd7\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-trusted-ca\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.276908 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49dffd22-1de1-4924-b13d-e0a406639b6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.277193 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.277193 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-installation-pull-secrets\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.277193 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.276981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-tls\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.277193 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.277080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.278298 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.278128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/49dffd22-1de1-4924-b13d-e0a406639b6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.279794 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.279749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-ca\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.280126 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.280102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.280438 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.280417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9775ec41-e21d-4485-a89e-8c489a3d26a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.280838 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.280813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.281310 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.281291 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/49dffd22-1de1-4924-b13d-e0a406639b6b-hub\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.285408 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.285371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpd7\" (UniqueName: \"kubernetes.io/projected/49dffd22-1de1-4924-b13d-e0a406639b6b-kube-api-access-krpd7\") pod \"cluster-proxy-proxy-agent-558dc54969-2l4nn\" (UID: \"49dffd22-1de1-4924-b13d-e0a406639b6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.285642 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.285622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vk2\" (UniqueName: \"kubernetes.io/projected/9775ec41-e21d-4485-a89e-8c489a3d26a4-kube-api-access-c7vk2\") pod \"managed-serviceaccount-addon-agent-5f94cd7696-mg8bw\" (UID: \"9775ec41-e21d-4485-a89e-8c489a3d26a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.375881 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.375796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" Jul 28 15:34:01.378123 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-trusted-ca\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378123 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-installation-pull-secrets\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378315 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-tls\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378315 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-bound-sa-token\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378315 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vng\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-kube-api-access-72vng\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378315 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-ca-trust-extracted\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378315 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-certificates\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.378589 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-image-registry-private-configuration\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.379006 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.378952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-ca-trust-extracted\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.379268 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.379243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-certificates\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.379268 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.379250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-trusted-ca\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.380947 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.380923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-installation-pull-secrets\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.381274 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.381241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-image-registry-private-configuration\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.381314 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.381287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-registry-tls\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.384072 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.384053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" Jul 28 15:34:01.386617 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.386594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-bound-sa-token\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.386753 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.386731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vng\" (UniqueName: \"kubernetes.io/projected/41b39437-1f55-4ca0-8b3d-dafd0f6570bf-kube-api-access-72vng\") pod \"image-registry-85cdd7bbb8-5886g\" (UID: \"41b39437-1f55-4ca0-8b3d-dafd0f6570bf\") " pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:01.474154 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:01.474131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:02.502552 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:02.502521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw"] Jul 28 15:34:02.535651 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:02.535622 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgsxf" Jul 28 15:34:02.547109 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:02.546974 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9775ec41_e21d_4485_a89e_8c489a3d26a4.slice/crio-3797e54e6c7e2403c8bcebc791f9c9471d4cd47963f3b86177106b19e6765cb7 WatchSource:0}: Error finding container 3797e54e6c7e2403c8bcebc791f9c9471d4cd47963f3b86177106b19e6765cb7: Status 404 returned error can't find the container with id 3797e54e6c7e2403c8bcebc791f9c9471d4cd47963f3b86177106b19e6765cb7 Jul 28 15:34:02.629285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:02.629215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" event={"ID":"9775ec41-e21d-4485-a89e-8c489a3d26a4","Type":"ContainerStarted","Data":"3797e54e6c7e2403c8bcebc791f9c9471d4cd47963f3b86177106b19e6765cb7"} Jul 28 15:34:02.675739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:02.675704 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn"] Jul 28 15:34:02.696763 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:02.696726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cdd7bbb8-5886g"] Jul 28 15:34:02.796087 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:02.796051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49dffd22_1de1_4924_b13d_e0a406639b6b.slice/crio-3ea4df3288c9fbd3a7df8a53fd5aec234ab33eff3efb079530bb93421dcdd64a WatchSource:0}: Error finding container 3ea4df3288c9fbd3a7df8a53fd5aec234ab33eff3efb079530bb93421dcdd64a: Status 404 returned error can't find the container with id 3ea4df3288c9fbd3a7df8a53fd5aec234ab33eff3efb079530bb93421dcdd64a Jul 28 15:34:02.796799 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:02.796776 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b39437_1f55_4ca0_8b3d_dafd0f6570bf.slice/crio-d87829b533f2a418bc28f92216cc832610d0ef47508dc43cd44f670c1d3f6cae WatchSource:0}: Error finding container d87829b533f2a418bc28f92216cc832610d0ef47508dc43cd44f670c1d3f6cae: Status 404 returned error can't find the container with id d87829b533f2a418bc28f92216cc832610d0ef47508dc43cd44f670c1d3f6cae Jul 28 15:34:03.634810 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.634764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zk6l7" event={"ID":"f5c0f11b-9b4d-4024-ae55-e56eb381c5da","Type":"ContainerStarted","Data":"33c3e80d707c13f855cbd31a7a88a80d2034aae9aad234d672403042d5a64f2c"} Jul 28 15:34:03.636015 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.635989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" event={"ID":"49dffd22-1de1-4924-b13d-e0a406639b6b","Type":"ContainerStarted","Data":"3ea4df3288c9fbd3a7df8a53fd5aec234ab33eff3efb079530bb93421dcdd64a"} Jul 28 15:34:03.637514 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.637489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" event={"ID":"41b39437-1f55-4ca0-8b3d-dafd0f6570bf","Type":"ContainerStarted","Data":"d766fa6cf0ebaa8c6d9c0615381f1598a27f70991635540622b90f8637c4e566"} Jul 28 15:34:03.637624 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.637530 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" event={"ID":"41b39437-1f55-4ca0-8b3d-dafd0f6570bf","Type":"ContainerStarted","Data":"d87829b533f2a418bc28f92216cc832610d0ef47508dc43cd44f670c1d3f6cae"} Jul 28 15:34:03.637681 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.637649 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:03.658784 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:03.658722 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" podStartSLOduration=2.658699358 podStartE2EDuration="2.658699358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:03.65767824 +0000 UTC m=+61.858525731" watchObservedRunningTime="2026-07-28 15:34:03.658699358 +0000 UTC m=+61.859546851" Jul 28 15:34:06.610446 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:06.610417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kzq4z" Jul 28 15:34:08.033274 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.033217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:34:08.035744 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.035717 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 15:34:08.046807 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.046782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43a7b6a7-b8df-412b-82fa-ba25e8ea512f-original-pull-secret\") pod \"global-pull-secret-syncer-wgwzd\" (UID: \"43a7b6a7-b8df-412b-82fa-ba25e8ea512f\") " pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:34:08.134449 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.134406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:34:08.134640 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.134501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:34:08.136864 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.136842 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.137030 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.136915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 15:34:08.147872 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.147837 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.148273 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.148241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30fe3678-fed3-49a2-8b35-1eed25e9c327-metrics-certs\") pod \"network-metrics-daemon-g965z\" (UID: \"30fe3678-fed3-49a2-8b35-1eed25e9c327\") " pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:34:08.158716 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.158682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmfj\" (UniqueName: \"kubernetes.io/projected/505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27-kube-api-access-rlmfj\") pod \"network-check-target-7j2v2\" (UID: \"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27\") " pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:34:08.283214 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.283179 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djz96\"" Jul 28 15:34:08.285290 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.285224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgwzd" Jul 28 15:34:08.290735 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.290710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:34:08.394054 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.394015 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hctgx\"" Jul 28 15:34:08.402735 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:08.402695 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g965z" Jul 28 15:34:09.554514 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.553675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7j2v2"] Jul 28 15:34:09.558480 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:09.558366 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505bfc5b_a2ad_4258_b7cd_d2c8ee8b4c27.slice/crio-c0537ca6405f2f7e3c2a75af3878ca59bb4ca6573213d4c479ebcbbc1eb8194d WatchSource:0}: Error finding container c0537ca6405f2f7e3c2a75af3878ca59bb4ca6573213d4c479ebcbbc1eb8194d: Status 404 returned error can't find the container with id c0537ca6405f2f7e3c2a75af3878ca59bb4ca6573213d4c479ebcbbc1eb8194d Jul 28 15:34:09.579157 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.578758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgwzd"] Jul 28 15:34:09.580067 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.579997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g965z"] Jul 28 15:34:09.583091 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:09.583063 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a7b6a7_b8df_412b_82fa_ba25e8ea512f.slice/crio-ab04daf4914a8d2fb93277260ec7040aa72586b612eb1cd35ec3f4edba9746c3 WatchSource:0}: Error finding container ab04daf4914a8d2fb93277260ec7040aa72586b612eb1cd35ec3f4edba9746c3: Status 404 returned error can't find the container with id ab04daf4914a8d2fb93277260ec7040aa72586b612eb1cd35ec3f4edba9746c3 Jul 28 15:34:09.583689 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:09.583671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fe3678_fed3_49a2_8b35_1eed25e9c327.slice/crio-00b1d0d8aaf375df74d6e050a17783d84554801b1e8ba30fe557ed8e589604a0 WatchSource:0}: Error finding container 00b1d0d8aaf375df74d6e050a17783d84554801b1e8ba30fe557ed8e589604a0: Status 404 returned error can't find the container with id 00b1d0d8aaf375df74d6e050a17783d84554801b1e8ba30fe557ed8e589604a0 Jul 28 15:34:09.662911 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.662879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zk6l7" event={"ID":"f5c0f11b-9b4d-4024-ae55-e56eb381c5da","Type":"ContainerStarted","Data":"f8292a912361878b53e2c0fb221f2c2d8956fa65a5d1971ba95fb987f80aa499"} Jul 28 15:34:09.664485 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.664445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" event={"ID":"49dffd22-1de1-4924-b13d-e0a406639b6b","Type":"ContainerStarted","Data":"3ad46c74a5987c798e9ad7741a922449e3d9b4e9780fcdf0e27e94180a08ae09"} Jul 28 15:34:09.665511 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.665488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g965z" event={"ID":"30fe3678-fed3-49a2-8b35-1eed25e9c327","Type":"ContainerStarted","Data":"00b1d0d8aaf375df74d6e050a17783d84554801b1e8ba30fe557ed8e589604a0"} Jul 28 15:34:09.666591 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.666564 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgwzd" event={"ID":"43a7b6a7-b8df-412b-82fa-ba25e8ea512f","Type":"ContainerStarted","Data":"ab04daf4914a8d2fb93277260ec7040aa72586b612eb1cd35ec3f4edba9746c3"} Jul 28 15:34:09.670606 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.670577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" event={"ID":"9775ec41-e21d-4485-a89e-8c489a3d26a4","Type":"ContainerStarted","Data":"837576b44b8a0cdb5469ee1da56d3653849c9f99b070abf8f18ca3cc1ec9a58f"} Jul 28 15:34:09.671630 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.671613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7j2v2" event={"ID":"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27","Type":"ContainerStarted","Data":"c0537ca6405f2f7e3c2a75af3878ca59bb4ca6573213d4c479ebcbbc1eb8194d"} Jul 28 15:34:09.684818 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.684778 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zk6l7" podStartSLOduration=17.200392099 podStartE2EDuration="32.684763177s" totalImagesPullingTime="15.484371078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:09.684045252 +0000 UTC m=+67.884892744" watchObservedRunningTime="2026-07-28 15:34:09.684763177 +0000 UTC m=+67.885610660" Jul 28 15:34:09.700184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:09.700137 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f94cd7696-mg8bw" podStartSLOduration=1.8454347100000001 podStartE2EDuration="8.700117623s" totalImagesPullingTime="6.854682913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:09.699872965 +0000 UTC m=+67.900720455" watchObservedRunningTime="2026-07-28 15:34:09.700117623 +0000 UTC m=+67.900965113" Jul 28 15:34:10.681530 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:10.681478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" event={"ID":"49dffd22-1de1-4924-b13d-e0a406639b6b","Type":"ContainerStarted","Data":"33e1ec58db1f27e25bf6a8a46b4018741a946dcf40a2b4684f283b4166398078"} Jul 28 15:34:10.682006 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:10.681547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" event={"ID":"49dffd22-1de1-4924-b13d-e0a406639b6b","Type":"ContainerStarted","Data":"8fb433f51375dd496551fc727738308298781e55c8691c8c95e33c8009731158"} Jul 28 15:34:10.702891 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:10.701872 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-558dc54969-2l4nn" podStartSLOduration=3.106020033 podStartE2EDuration="9.701848791s" totalImagesPullingTime="6.595828758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.701208482 +0000 UTC m=+68.902055974" watchObservedRunningTime="2026-07-28 15:34:10.701848791 +0000 UTC m=+68.902696284" Jul 28 15:34:11.075146 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:11.075079 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:34:11.687952 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:11.687871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g965z" event={"ID":"30fe3678-fed3-49a2-8b35-1eed25e9c327","Type":"ContainerStarted","Data":"b05abdf37fe1aadfdcc85eb13dd2b8b3d3b3fa3476405420b95616142bc562db"} Jul 28 15:34:11.687952 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:11.687937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g965z" event={"ID":"30fe3678-fed3-49a2-8b35-1eed25e9c327","Type":"ContainerStarted","Data":"026951e6d358fe5a9319f70b68e204d9f905b89d79f379b88f0ed4de83e48246"} Jul 28 15:34:11.708276 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:11.708219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g965z" podStartSLOduration=68.040215893 podStartE2EDuration="1m9.708202633s" totalImagesPullingTime="1.66798674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:11.707611874 +0000 UTC m=+69.908459367" watchObservedRunningTime="2026-07-28 15:34:11.708202633 +0000 UTC m=+69.909050129" Jul 28 15:34:14.702099 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:14.702057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7j2v2" event={"ID":"505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27","Type":"ContainerStarted","Data":"6021c42d4aac1998931e97817e6ff4625cdddb41d95083ff044ecd617f9af40b"} Jul 28 15:34:14.702678 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:14.702224 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:34:14.703510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:14.703490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgwzd" event={"ID":"43a7b6a7-b8df-412b-82fa-ba25e8ea512f","Type":"ContainerStarted","Data":"6aff4e70c27e5393a7923e5bdd44d3ffc1407bb221d7ad29f01821391f2f825d"} Jul 28 15:34:14.726409 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:14.726324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7j2v2" podStartSLOduration=68.178356525 podStartE2EDuration="1m12.726304866s" totalImagesPullingTime="4.547948341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:14.725727013 +0000 UTC m=+72.926574505" watchObservedRunningTime="2026-07-28 15:34:14.726304866 +0000 UTC m=+72.927152359" Jul 28 15:34:14.740958 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:14.740910 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wgwzd" podStartSLOduration=68.215106832 podStartE2EDuration="1m12.740897434s" totalImagesPullingTime="4.525790602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:14.740157517 +0000 UTC m=+72.941005007" watchObservedRunningTime="2026-07-28 15:34:14.740897434 +0000 UTC m=+72.941744925" Jul 28 15:34:18.017300 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.017261 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tk8vw"] Jul 28 15:34:18.020058 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.020034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.021831 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.021803 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 15:34:18.021960 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.021890 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t8s9r\"" Jul 28 15:34:18.022092 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.022079 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 15:34:18.022274 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.022253 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 15:34:18.022546 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.022530 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 15:34:18.022643 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.022596 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 15:34:18.022702 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.022651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 15:34:18.114483 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-textfile\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114483 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-tls\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9z79\" (UniqueName: \"kubernetes.io/projected/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-kube-api-access-t9z79\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-metrics-client-ca\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-wtmp\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-root\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.114732 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.114672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-sys\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215473 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-tls\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215473 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9z79\" (UniqueName: \"kubernetes.io/projected/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-kube-api-access-t9z79\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215677 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215677 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-metrics-client-ca\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-wtmp\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215825 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-root\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215825 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-sys\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215918 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-textfile\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215918 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-sys\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215918 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-root\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.215918 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.215873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-wtmp\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.216114 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.216083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-textfile\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.216192 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.216168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-metrics-client-ca\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.216308 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.216289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.217880 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.217850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.217983 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.217929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-node-exporter-tls\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.223296 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.223272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9z79\" (UniqueName: \"kubernetes.io/projected/2ef38ff7-f7b9-4935-a387-6ae9f0e25542-kube-api-access-t9z79\") pod \"node-exporter-tk8vw\" (UID: \"2ef38ff7-f7b9-4935-a387-6ae9f0e25542\") " pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.330631 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.330546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tk8vw" Jul 28 15:34:18.342577 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:18.342529 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef38ff7_f7b9_4935_a387_6ae9f0e25542.slice/crio-ca364f53befdac171d2de925a201f8620dd65591acd5a97c3e36ec5531f21c9a WatchSource:0}: Error finding container ca364f53befdac171d2de925a201f8620dd65591acd5a97c3e36ec5531f21c9a: Status 404 returned error can't find the container with id ca364f53befdac171d2de925a201f8620dd65591acd5a97c3e36ec5531f21c9a Jul 28 15:34:18.718444 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:18.718341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk8vw" event={"ID":"2ef38ff7-f7b9-4935-a387-6ae9f0e25542","Type":"ContainerStarted","Data":"ca364f53befdac171d2de925a201f8620dd65591acd5a97c3e36ec5531f21c9a"} Jul 28 15:34:19.723914 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:19.723881 2575 generic.go:356] "Generic (PLEG): container finished" podID="2ef38ff7-f7b9-4935-a387-6ae9f0e25542" containerID="2bd794384c74fbe25a0a8d36851c3a725da69bd7a7a915adfd7c876afc3818f3" exitCode=0 Jul 28 15:34:19.724297 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:19.723969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk8vw" event={"ID":"2ef38ff7-f7b9-4935-a387-6ae9f0e25542","Type":"ContainerDied","Data":"2bd794384c74fbe25a0a8d36851c3a725da69bd7a7a915adfd7c876afc3818f3"} Jul 28 15:34:20.729159 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:20.729119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk8vw" event={"ID":"2ef38ff7-f7b9-4935-a387-6ae9f0e25542","Type":"ContainerStarted","Data":"ba2b20b0a0a83c5f49892f15aa80dc94a603231747207f7d58de77707f4508c7"} Jul 28 15:34:20.729657 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:20.729172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk8vw" event={"ID":"2ef38ff7-f7b9-4935-a387-6ae9f0e25542","Type":"ContainerStarted","Data":"6a1ec6ab5ca4aa8c6a7fa23cab7e6bab0c87d9f6a44d9affb28dfe5b444689e5"} Jul 28 15:34:20.770753 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:20.770697 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tk8vw" podStartSLOduration=2.845025465 podStartE2EDuration="3.770680519s" totalImagesPullingTime="859.999329ms" totalInitContainerRuntime="65.655725ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:20.770374723 +0000 UTC m=+78.971222213" watchObservedRunningTime="2026-07-28 15:34:20.770680519 +0000 UTC m=+78.971528010" Jul 28 15:34:24.648140 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:24.648010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85cdd7bbb8-5886g" Jul 28 15:34:26.090268 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.090231 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" podUID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" containerName="registry" containerID="cri-o://3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519" gracePeriod=30 Jul 28 15:34:26.329607 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.329583 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:34:26.481294 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481203 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481294 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh2c2\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481294 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481264 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481294 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481644 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481372 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481644 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481644 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481644 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481561 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca\") pod \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\" (UID: \"cc032a8d-a6ff-472e-9dba-d7f2451198ce\") " Jul 28 15:34:26.481870 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.481844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:34:26.482119 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.482082 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:34:26.484155 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.484130 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:34:26.484266 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.484189 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2" (OuterVolumeSpecName: "kube-api-access-jh2c2") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "kube-api-access-jh2c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:34:26.484266 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.484209 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:34:26.484382 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.484258 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:34:26.484382 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.484317 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:34:26.490168 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.490143 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cc032a8d-a6ff-472e-9dba-d7f2451198ce" (UID: "cc032a8d-a6ff-472e-9dba-d7f2451198ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:34:26.582120 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582068 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582120 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582110 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-trusted-ca\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582120 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582120 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-image-registry-private-configuration\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582120 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582129 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2c2\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-kube-api-access-jh2c2\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582120 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582141 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc032a8d-a6ff-472e-9dba-d7f2451198ce-bound-sa-token\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582468 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582150 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc032a8d-a6ff-472e-9dba-d7f2451198ce-ca-trust-extracted\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582468 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582158 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc032a8d-a6ff-472e-9dba-d7f2451198ce-registry-certificates\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.582468 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.582166 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc032a8d-a6ff-472e-9dba-d7f2451198ce-installation-pull-secrets\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:34:26.751790 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.751703 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" containerID="3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519" exitCode=0 Jul 28 15:34:26.751790 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.751753 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" Jul 28 15:34:26.751947 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.751794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" event={"ID":"cc032a8d-a6ff-472e-9dba-d7f2451198ce","Type":"ContainerDied","Data":"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519"} Jul 28 15:34:26.751947 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.751849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4f76c666-5pk7c" event={"ID":"cc032a8d-a6ff-472e-9dba-d7f2451198ce","Type":"ContainerDied","Data":"12aa8ca8bd6a229ba8587f20ee1b71e72a166a6e5f48066df7153045c6587679"} Jul 28 15:34:26.751947 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.751869 2575 scope.go:117] "RemoveContainer" containerID="3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519" Jul 28 15:34:26.760144 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.760124 2575 scope.go:117] "RemoveContainer" containerID="3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519" Jul 28 15:34:26.760437 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:34:26.760418 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519\": container with ID starting with 3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519 not found: ID does not exist" containerID="3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519" Jul 28 15:34:26.760480 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.760445 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519"} err="failed to get container status \"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519\": rpc error: code = NotFound desc = could not find container \"3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519\": container with ID starting with 3479dc056e6475fb4ae46d560e3b1e074fb92c54ee22d9b1b76c6ed06961b519 not found: ID does not exist" Jul 28 15:34:26.774818 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.774789 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:34:26.782988 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:26.782965 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b4f76c666-5pk7c"] Jul 28 15:34:28.258700 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.258666 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:34:28.259061 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.258922 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" containerName="registry" Jul 28 15:34:28.259061 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.258934 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" containerName="registry" Jul 28 15:34:28.259061 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.258993 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" containerName="registry" Jul 28 15:34:28.266441 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.266415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.268780 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.268743 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 15:34:28.268927 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.268747 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 15:34:28.269381 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269364 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 15:34:28.269511 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 15:34:28.269575 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269531 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 15:34:28.269626 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269580 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dpkp2\"" Jul 28 15:34:28.269774 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269745 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 15:34:28.269836 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.269810 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 15:34:28.272071 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.272048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:34:28.274933 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.274914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 15:34:28.372751 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.372717 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc032a8d-a6ff-472e-9dba-d7f2451198ce" path="/var/lib/kubelet/pods/cc032a8d-a6ff-472e-9dba-d7f2451198ce/volumes" Jul 28 15:34:28.395221 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395221 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fjz\" (UniqueName: \"kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.395440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.395354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496289 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496289 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496525 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496525 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496525 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fjz\" (UniqueName: \"kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496672 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.496672 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.496589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.497177 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.497142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.497299 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.497276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.497367 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.497339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.497465 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.497449 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.498849 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.498822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.499017 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.499000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.505037 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.505005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fjz\" (UniqueName: \"kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz\") pod \"console-5f79cfd86d-q7hmx\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.577095 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.577008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:28.722140 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.722107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:34:28.726206 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:34:28.726181 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98af8e9_b696_4f04_8148_7fc5c8cd6212.slice/crio-782798c736e8767ace6fdb1538ac99ddd75bcc3d214b5d2f0058b98ec7863d6b WatchSource:0}: Error finding container 782798c736e8767ace6fdb1538ac99ddd75bcc3d214b5d2f0058b98ec7863d6b: Status 404 returned error can't find the container with id 782798c736e8767ace6fdb1538ac99ddd75bcc3d214b5d2f0058b98ec7863d6b Jul 28 15:34:28.761342 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:28.761309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f79cfd86d-q7hmx" event={"ID":"b98af8e9-b696-4f04-8148-7fc5c8cd6212","Type":"ContainerStarted","Data":"782798c736e8767ace6fdb1538ac99ddd75bcc3d214b5d2f0058b98ec7863d6b"} Jul 28 15:34:32.779673 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:32.779634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f79cfd86d-q7hmx" event={"ID":"b98af8e9-b696-4f04-8148-7fc5c8cd6212","Type":"ContainerStarted","Data":"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0"} Jul 28 15:34:32.797013 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:32.796953 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f79cfd86d-q7hmx" podStartSLOduration=1.084389306 podStartE2EDuration="4.796931374s" totalImagesPullingTime="3.712542068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:32.796840374 +0000 UTC m=+90.997687866" watchObservedRunningTime="2026-07-28 15:34:32.796931374 +0000 UTC m=+90.997778865" Jul 28 15:34:38.577779 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:38.577740 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:38.578233 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:38.577800 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:38.582722 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:38.582700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:38.804184 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:38.804158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:34:45.709669 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:34:45.709639 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7j2v2" Jul 28 15:35:45.844327 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.844290 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:35:45.848799 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.848781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.858531 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.858503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:35:45.962292 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962510 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:45.962701 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:45.962534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5rx\" (UniqueName: \"kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063168 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063168 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063545 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063545 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063545 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5rx\" (UniqueName: \"kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063545 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063768 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.063991 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.063965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.064092 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.064026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.064212 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.064190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.064382 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.064362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.065771 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.065754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.065881 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.065865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.075305 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.075285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5rx\" (UniqueName: \"kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx\") pod \"console-55cfb8655-kv8wp\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.158767 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.158678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:35:46.301179 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:46.301149 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:35:46.304733 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:35:46.304701 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee44d1b7_b488_4708_a097_c965b8688a4c.slice/crio-4b63512aa67e15fa3f3f884652b2c72647f9bf9ced4bfebfb365108cec278dca WatchSource:0}: Error finding container 4b63512aa67e15fa3f3f884652b2c72647f9bf9ced4bfebfb365108cec278dca: Status 404 returned error can't find the container with id 4b63512aa67e15fa3f3f884652b2c72647f9bf9ced4bfebfb365108cec278dca Jul 28 15:35:47.030222 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:47.030185 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cfb8655-kv8wp" event={"ID":"ee44d1b7-b488-4708-a097-c965b8688a4c","Type":"ContainerStarted","Data":"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e"} Jul 28 15:35:47.030728 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:47.030244 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cfb8655-kv8wp" event={"ID":"ee44d1b7-b488-4708-a097-c965b8688a4c","Type":"ContainerStarted","Data":"4b63512aa67e15fa3f3f884652b2c72647f9bf9ced4bfebfb365108cec278dca"} Jul 28 15:35:47.049950 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:47.049896 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55cfb8655-kv8wp" podStartSLOduration=2.049878166 podStartE2EDuration="2.049878166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:47.048785386 +0000 UTC m=+165.249632878" watchObservedRunningTime="2026-07-28 15:35:47.049878166 +0000 UTC m=+165.250725657" Jul 28 15:35:53.352052 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.352015 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:35:53.381643 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.381606 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:35:53.384515 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.384494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.396545 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.396520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:35:53.525693 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525693 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbxf\" (UniqueName: \"kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525693 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525693 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525956 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525956 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.525956 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.525827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.626999 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.626921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.626999 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.626955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.626999 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.626979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627267 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627267 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627267 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627267 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbxf\" (UniqueName: \"kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627737 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627843 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.627882 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.627823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.628035 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.628018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.629913 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.629886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.630029 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.630009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.636876 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.636857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbxf\" (UniqueName: \"kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf\") pod \"console-6659746ccd-tcskk\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.693611 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.693565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:35:53.823885 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:53.823853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:35:53.827058 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:35:53.827028 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb334e833_bfd5_43f4_becd_1b2db5941425.slice/crio-4f5456e34c5bc32707f1fe3452c479380d4f1519df9715c5ff4f7bfb950d3a28 WatchSource:0}: Error finding container 4f5456e34c5bc32707f1fe3452c479380d4f1519df9715c5ff4f7bfb950d3a28: Status 404 returned error can't find the container with id 4f5456e34c5bc32707f1fe3452c479380d4f1519df9715c5ff4f7bfb950d3a28 Jul 28 15:35:54.054955 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:54.054915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6659746ccd-tcskk" event={"ID":"b334e833-bfd5-43f4-becd-1b2db5941425","Type":"ContainerStarted","Data":"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01"} Jul 28 15:35:54.055138 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:54.054994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6659746ccd-tcskk" event={"ID":"b334e833-bfd5-43f4-becd-1b2db5941425","Type":"ContainerStarted","Data":"4f5456e34c5bc32707f1fe3452c479380d4f1519df9715c5ff4f7bfb950d3a28"} Jul 28 15:35:54.072414 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:54.072339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6659746ccd-tcskk" podStartSLOduration=1.072324217 podStartE2EDuration="1.072324217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:54.070933757 +0000 UTC m=+172.271781249" watchObservedRunningTime="2026-07-28 15:35:54.072324217 +0000 UTC m=+172.273171707" Jul 28 15:35:56.159039 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:35:56.158998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:36:03.693968 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:03.693930 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:36:03.694447 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:03.694202 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:36:03.698868 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:03.698850 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:36:04.096904 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:04.096873 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:36:04.140084 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:04.140051 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:36:18.370952 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.370886 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55cfb8655-kv8wp" podUID="ee44d1b7-b488-4708-a097-c965b8688a4c" containerName="console" containerID="cri-o://8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e" gracePeriod=15 Jul 28 15:36:18.609660 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.609636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:36:18.727385 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727300 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727385 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727354 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp5rx\" (UniqueName: \"kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx\") pod \"ee44d1b7-b488-4708-a097-c965b8688a4c\" (UID: \"ee44d1b7-b488-4708-a097-c965b8688a4c\") " Jul 28 15:36:18.727878 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727818 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:18.727926 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727873 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:18.727966 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config" (OuterVolumeSpecName: "console-config") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:18.727966 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.727952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:18.729928 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.729892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:36:18.729928 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.729914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:36:18.730060 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.729946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx" (OuterVolumeSpecName: "kube-api-access-lp5rx") pod "ee44d1b7-b488-4708-a097-c965b8688a4c" (UID: "ee44d1b7-b488-4708-a097-c965b8688a4c"). InnerVolumeSpecName "kube-api-access-lp5rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:36:18.828486 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828454 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828486 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828480 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee44d1b7-b488-4708-a097-c965b8688a4c-console-oauth-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828486 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828489 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp5rx\" (UniqueName: \"kubernetes.io/projected/ee44d1b7-b488-4708-a097-c965b8688a4c-kube-api-access-lp5rx\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828713 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828501 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-trusted-ca-bundle\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828713 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828509 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-service-ca\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828713 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828517 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-oauth-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:18.828713 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:18.828525 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee44d1b7-b488-4708-a097-c965b8688a4c-console-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:19.143002 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.142972 2575 generic.go:356] "Generic (PLEG): container finished" podID="ee44d1b7-b488-4708-a097-c965b8688a4c" containerID="8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e" exitCode=0 Jul 28 15:36:19.143169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.143017 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cfb8655-kv8wp" Jul 28 15:36:19.143169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.143034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cfb8655-kv8wp" event={"ID":"ee44d1b7-b488-4708-a097-c965b8688a4c","Type":"ContainerDied","Data":"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e"} Jul 28 15:36:19.143169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.143076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cfb8655-kv8wp" event={"ID":"ee44d1b7-b488-4708-a097-c965b8688a4c","Type":"ContainerDied","Data":"4b63512aa67e15fa3f3f884652b2c72647f9bf9ced4bfebfb365108cec278dca"} Jul 28 15:36:19.143169 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.143096 2575 scope.go:117] "RemoveContainer" containerID="8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e" Jul 28 15:36:19.151207 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.151186 2575 scope.go:117] "RemoveContainer" containerID="8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e" Jul 28 15:36:19.151505 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:36:19.151487 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e\": container with ID starting with 8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e not found: ID does not exist" containerID="8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e" Jul 28 15:36:19.151585 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.151513 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e"} err="failed to get container status \"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e\": rpc error: code = NotFound desc = could not find container \"8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e\": container with ID starting with 8d819876c9565a1e76ab58fb2684507b0599a84549c9cb0e52744b4008013f9e not found: ID does not exist" Jul 28 15:36:19.163340 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.163313 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:36:19.167221 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:19.167200 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55cfb8655-kv8wp"] Jul 28 15:36:20.372301 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:20.372267 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee44d1b7-b488-4708-a097-c965b8688a4c" path="/var/lib/kubelet/pods/ee44d1b7-b488-4708-a097-c965b8688a4c/volumes" Jul 28 15:36:29.159454 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.159370 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f79cfd86d-q7hmx" podUID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" containerName="console" containerID="cri-o://d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0" gracePeriod=15 Jul 28 15:36:29.405323 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.405292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:36:29.510811 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510772 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.510994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510830 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.510994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.510994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510891 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9fjz\" (UniqueName: \"kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.510994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510921 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.510994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.510966 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.511256 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.511021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config\") pod \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\" (UID: \"b98af8e9-b696-4f04-8148-7fc5c8cd6212\") " Jul 28 15:36:29.511310 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.511241 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca" (OuterVolumeSpecName: "service-ca") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:29.511310 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.511261 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config" (OuterVolumeSpecName: "console-config") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:29.511438 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.511306 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:29.511664 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.511640 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:36:29.513505 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.513411 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:36:29.513505 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.513462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:36:29.513679 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.513534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz" (OuterVolumeSpecName: "kube-api-access-x9fjz") pod "b98af8e9-b696-4f04-8148-7fc5c8cd6212" (UID: "b98af8e9-b696-4f04-8148-7fc5c8cd6212"). InnerVolumeSpecName "kube-api-access-x9fjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:36:29.611805 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611768 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.611805 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611801 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fjz\" (UniqueName: \"kubernetes.io/projected/b98af8e9-b696-4f04-8148-7fc5c8cd6212-kube-api-access-x9fjz\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.611805 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611812 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-oauth-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.612025 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611821 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.612025 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611829 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b98af8e9-b696-4f04-8148-7fc5c8cd6212-console-oauth-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.612025 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611837 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-service-ca\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:29.612025 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:29.611845 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98af8e9-b696-4f04-8148-7fc5c8cd6212-trusted-ca-bundle\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:36:30.189476 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.189437 2575 generic.go:356] "Generic (PLEG): container finished" podID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" containerID="d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0" exitCode=0 Jul 28 15:36:30.189862 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.189488 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f79cfd86d-q7hmx" Jul 28 15:36:30.189862 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.189516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f79cfd86d-q7hmx" event={"ID":"b98af8e9-b696-4f04-8148-7fc5c8cd6212","Type":"ContainerDied","Data":"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0"} Jul 28 15:36:30.189862 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.189565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f79cfd86d-q7hmx" event={"ID":"b98af8e9-b696-4f04-8148-7fc5c8cd6212","Type":"ContainerDied","Data":"782798c736e8767ace6fdb1538ac99ddd75bcc3d214b5d2f0058b98ec7863d6b"} Jul 28 15:36:30.189862 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.189593 2575 scope.go:117] "RemoveContainer" containerID="d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0" Jul 28 15:36:30.197480 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.197456 2575 scope.go:117] "RemoveContainer" containerID="d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0" Jul 28 15:36:30.197761 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:36:30.197737 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0\": container with ID starting with d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0 not found: ID does not exist" containerID="d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0" Jul 28 15:36:30.197819 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.197770 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0"} err="failed to get container status \"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0\": rpc error: code = NotFound desc = could not find container \"d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0\": container with ID starting with d7223e5972e4f20412b1d2aaf4fb3ef450d99b0fb065d76e1463a2a0cb879eb0 not found: ID does not exist" Jul 28 15:36:30.209628 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.209600 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:36:30.213366 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.213347 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f79cfd86d-q7hmx"] Jul 28 15:36:30.371971 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:36:30.371940 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" path="/var/lib/kubelet/pods/b98af8e9-b696-4f04-8148-7fc5c8cd6212/volumes" Jul 28 15:37:38.350426 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350339 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb"] Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350624 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee44d1b7-b488-4708-a097-c965b8688a4c" containerName="console" Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350637 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee44d1b7-b488-4708-a097-c965b8688a4c" containerName="console" Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350654 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" containerName="console" Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350660 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" containerName="console" Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350699 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee44d1b7-b488-4708-a097-c965b8688a4c" containerName="console" Jul 28 15:37:38.350874 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.350707 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98af8e9-b696-4f04-8148-7fc5c8cd6212" containerName="console" Jul 28 15:37:38.353565 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.353546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.355563 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.355542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 15:37:38.355672 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.355658 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 15:37:38.356219 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.356204 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npt6t\"" Jul 28 15:37:38.361139 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.361116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb"] Jul 28 15:37:38.393343 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.393310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.393343 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.393344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmpl\" (UniqueName: \"kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.393570 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.393364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.494068 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.494022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.494068 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.494066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmpl\" (UniqueName: \"kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.494277 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.494085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.494445 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.494428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.494492 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.494456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.502604 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.502576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmpl\" (UniqueName: \"kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.663171 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.663076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:38.791809 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:38.791776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb"] Jul 28 15:37:38.794765 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:37:38.794732 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c3a7_b514_4f33_bcca_15205cdd44f3.slice/crio-22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5 WatchSource:0}: Error finding container 22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5: Status 404 returned error can't find the container with id 22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5 Jul 28 15:37:39.415359 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:39.415323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" event={"ID":"51b1c3a7-b514-4f33-bcca-15205cdd44f3","Type":"ContainerStarted","Data":"22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5"} Jul 28 15:37:46.442428 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:46.442369 2575 generic.go:356] "Generic (PLEG): container finished" podID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerID="e2ba2cdf814bb8096572a5a5634df5113362d1339a3c440172a8f4c1ccce3984" exitCode=0 Jul 28 15:37:46.442821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:46.442461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" event={"ID":"51b1c3a7-b514-4f33-bcca-15205cdd44f3","Type":"ContainerDied","Data":"e2ba2cdf814bb8096572a5a5634df5113362d1339a3c440172a8f4c1ccce3984"} Jul 28 15:37:49.454860 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:49.454826 2575 generic.go:356] "Generic (PLEG): container finished" podID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerID="f787b6101ce050874be765fc775ca08cfaa32f18c263718920c0560d87640d86" exitCode=0 Jul 28 15:37:49.455345 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:49.454887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" event={"ID":"51b1c3a7-b514-4f33-bcca-15205cdd44f3","Type":"ContainerDied","Data":"f787b6101ce050874be765fc775ca08cfaa32f18c263718920c0560d87640d86"} Jul 28 15:37:57.485200 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:57.485160 2575 generic.go:356] "Generic (PLEG): container finished" podID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerID="30ec9df895cd7951d90bbdde0b0e9074863445a336c975577147f5ae5c18f0f9" exitCode=0 Jul 28 15:37:57.485613 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:57.485250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" event={"ID":"51b1c3a7-b514-4f33-bcca-15205cdd44f3","Type":"ContainerDied","Data":"30ec9df895cd7951d90bbdde0b0e9074863445a336c975577147f5ae5c18f0f9"} Jul 28 15:37:58.618143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.618117 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:58.656624 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.656589 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmpl\" (UniqueName: \"kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl\") pod \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " Jul 28 15:37:58.656786 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.656648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util\") pod \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " Jul 28 15:37:58.656786 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.656752 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle\") pod \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\" (UID: \"51b1c3a7-b514-4f33-bcca-15205cdd44f3\") " Jul 28 15:37:58.657433 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.657382 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle" (OuterVolumeSpecName: "bundle") pod "51b1c3a7-b514-4f33-bcca-15205cdd44f3" (UID: "51b1c3a7-b514-4f33-bcca-15205cdd44f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:37:58.658845 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.658818 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl" (OuterVolumeSpecName: "kube-api-access-dtmpl") pod "51b1c3a7-b514-4f33-bcca-15205cdd44f3" (UID: "51b1c3a7-b514-4f33-bcca-15205cdd44f3"). InnerVolumeSpecName "kube-api-access-dtmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:37:58.661400 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.661372 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util" (OuterVolumeSpecName: "util") pod "51b1c3a7-b514-4f33-bcca-15205cdd44f3" (UID: "51b1c3a7-b514-4f33-bcca-15205cdd44f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:37:58.757343 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.757311 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-bundle\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:37:58.757343 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.757345 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmpl\" (UniqueName: \"kubernetes.io/projected/51b1c3a7-b514-4f33-bcca-15205cdd44f3-kube-api-access-dtmpl\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:37:58.757549 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:58.757355 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51b1c3a7-b514-4f33-bcca-15205cdd44f3-util\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:37:59.493054 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:59.493016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" event={"ID":"51b1c3a7-b514-4f33-bcca-15205cdd44f3","Type":"ContainerDied","Data":"22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5"} Jul 28 15:37:59.493054 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:59.493044 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lmsgb" Jul 28 15:37:59.493296 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:37:59.493064 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ca99e6ce024f8972abd85dfda770a864411960438d00d883cc197ee3475ae5" Jul 28 15:38:02.190505 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:38:02.190480 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 15:39:29.025984 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.025946 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-5q6mw"] Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026323 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="extract" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026337 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="extract" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026351 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="util" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026361 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="util" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026377 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="pull" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026385 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="pull" Jul 28 15:39:29.026477 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.026457 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b1c3a7-b514-4f33-bcca-15205cdd44f3" containerName="extract" Jul 28 15:39:29.028487 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.028470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.030445 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.030417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 28 15:39:29.031414 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.031376 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 15:39:29.031696 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.031417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-8c4j6\"" Jul 28 15:39:29.031851 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.031461 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 15:39:29.036526 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.036505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-5q6mw"] Jul 28 15:39:29.098745 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.098710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59fg\" (UniqueName: \"kubernetes.io/projected/6fc9d1de-152f-4a06-9361-d1da024a57cd-kube-api-access-j59fg\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.098932 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.098764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.199235 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.199193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.199456 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.199272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59fg\" (UniqueName: \"kubernetes.io/projected/6fc9d1de-152f-4a06-9361-d1da024a57cd-kube-api-access-j59fg\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.199456 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:39:29.199342 2575 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 28 15:39:29.199456 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:39:29.199428 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert podName:6fc9d1de-152f-4a06-9361-d1da024a57cd nodeName:}" failed. No retries permitted until 2026-07-28 15:39:29.699406402 +0000 UTC m=+387.900253871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert") pod "llmisvc-controller-manager-84fc959947-5q6mw" (UID: "6fc9d1de-152f-4a06-9361-d1da024a57cd") : secret "llmisvc-webhook-server-cert" not found Jul 28 15:39:29.207513 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.207483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59fg\" (UniqueName: \"kubernetes.io/projected/6fc9d1de-152f-4a06-9361-d1da024a57cd-kube-api-access-j59fg\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.703550 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.703506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.705958 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.705928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fc9d1de-152f-4a06-9361-d1da024a57cd-cert\") pod \"llmisvc-controller-manager-84fc959947-5q6mw\" (UID: \"6fc9d1de-152f-4a06-9361-d1da024a57cd\") " pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:29.939901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:29.939859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:30.065577 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:30.065542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-5q6mw"] Jul 28 15:39:30.068585 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:39:30.068562 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6fc9d1de_152f_4a06_9361_d1da024a57cd.slice/crio-b218e90bb6dc9b0db34d3360f0bacbd654be9579f8c06fbd658c80fcfbed6769 WatchSource:0}: Error finding container b218e90bb6dc9b0db34d3360f0bacbd654be9579f8c06fbd658c80fcfbed6769: Status 404 returned error can't find the container with id b218e90bb6dc9b0db34d3360f0bacbd654be9579f8c06fbd658c80fcfbed6769 Jul 28 15:39:30.070275 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:30.070260 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:39:30.787963 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:30.787930 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" event={"ID":"6fc9d1de-152f-4a06-9361-d1da024a57cd","Type":"ContainerStarted","Data":"b218e90bb6dc9b0db34d3360f0bacbd654be9579f8c06fbd658c80fcfbed6769"} Jul 28 15:39:33.800371 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:33.800331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" event={"ID":"6fc9d1de-152f-4a06-9361-d1da024a57cd","Type":"ContainerStarted","Data":"58e7ab921352b99c201b2c897987852a30d714c960e49727c627d1aa7e910c49"} Jul 28 15:39:33.800760 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:33.800474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:39:33.822917 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:39:33.822867 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" podStartSLOduration=2.072963277 podStartE2EDuration="4.822852209s" totalImagesPullingTime="2.749888932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:39:33.822278335 +0000 UTC m=+392.023125827" watchObservedRunningTime="2026-07-28 15:39:33.822852209 +0000 UTC m=+392.023699699" Jul 28 15:40:04.805365 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:04.805332 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-5q6mw" Jul 28 15:40:45.820486 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.820386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zvg49"] Jul 28 15:40:45.828848 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.828806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.831253 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.831225 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 28 15:40:45.832152 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.832131 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4kgb5\"" Jul 28 15:40:45.832533 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.832512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zvg49"] Jul 28 15:40:45.876903 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.876856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjd4\" (UniqueName: \"kubernetes.io/projected/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-kube-api-access-nnjd4\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.877073 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.876911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-cert\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.978334 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.978292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjd4\" (UniqueName: \"kubernetes.io/projected/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-kube-api-access-nnjd4\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.978334 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.978340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-cert\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.980769 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.980749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-cert\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:45.986022 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:45.985999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjd4\" (UniqueName: \"kubernetes.io/projected/37ffa276-f1bc-4f01-af7c-8a88a2c7f5db-kube-api-access-nnjd4\") pod \"odh-model-controller-75d9cdbddf-zvg49\" (UID: \"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db\") " pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:46.141853 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:46.141752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:46.270284 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:46.270252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zvg49"] Jul 28 15:40:46.273406 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:40:46.273366 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ffa276_f1bc_4f01_af7c_8a88a2c7f5db.slice/crio-1430fffdfe0863972db33642812df15bb19ef4d4b74eb9b0f3b674946eeb9678 WatchSource:0}: Error finding container 1430fffdfe0863972db33642812df15bb19ef4d4b74eb9b0f3b674946eeb9678: Status 404 returned error can't find the container with id 1430fffdfe0863972db33642812df15bb19ef4d4b74eb9b0f3b674946eeb9678 Jul 28 15:40:47.049703 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:47.049647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" event={"ID":"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db","Type":"ContainerStarted","Data":"1430fffdfe0863972db33642812df15bb19ef4d4b74eb9b0f3b674946eeb9678"} Jul 28 15:40:50.064842 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:50.064801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" event={"ID":"37ffa276-f1bc-4f01-af7c-8a88a2c7f5db","Type":"ContainerStarted","Data":"8deb5674cdd10098eb0413330f4185849421200c763155cd87c70b453f5080f3"} Jul 28 15:40:50.065211 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:50.064920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:40:50.082436 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:50.082373 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" podStartSLOduration=2.151759411 podStartE2EDuration="5.082355257s" totalImagesPullingTime="2.930595846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:50.081214326 +0000 UTC m=+468.282061817" watchObservedRunningTime="2026-07-28 15:40:50.082355257 +0000 UTC m=+468.283202748" Jul 28 15:40:52.248868 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.248835 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66755c677b-tst75"] Jul 28 15:40:52.252036 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.252016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.262816 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.262778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66755c677b-tst75"] Jul 28 15:40:52.327433 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-oauth-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327609 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-service-ca\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327609 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-trusted-ca-bundle\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327609 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-oauth-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-console-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8tm\" (UniqueName: \"kubernetes.io/projected/dbabdc75-6227-492f-aede-af87d3ef8158-kube-api-access-nq8tm\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.327739 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.327662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.428939 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.428904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-oauth-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.428939 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.428949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-service-ca\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.428966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-trusted-ca-bundle\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.428988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-oauth-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-console-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8tm\" (UniqueName: \"kubernetes.io/projected/dbabdc75-6227-492f-aede-af87d3ef8158-kube-api-access-nq8tm\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429217 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429804 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429777 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-service-ca\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429895 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429777 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-console-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.429895 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.429871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-trusted-ca-bundle\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.430098 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.430075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dbabdc75-6227-492f-aede-af87d3ef8158-oauth-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.431647 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.431626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-serving-cert\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.431856 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.431833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dbabdc75-6227-492f-aede-af87d3ef8158-console-oauth-config\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.437467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.437439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8tm\" (UniqueName: \"kubernetes.io/projected/dbabdc75-6227-492f-aede-af87d3ef8158-kube-api-access-nq8tm\") pod \"console-66755c677b-tst75\" (UID: \"dbabdc75-6227-492f-aede-af87d3ef8158\") " pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.560302 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.560268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:40:52.694542 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:52.694510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66755c677b-tst75"] Jul 28 15:40:52.697698 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:40:52.697666 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbabdc75_6227_492f_aede_af87d3ef8158.slice/crio-8e8dd47f6c74e7d696efc16ffdbf9003f2fe8dd3630ccac4003ac066c32746d0 WatchSource:0}: Error finding container 8e8dd47f6c74e7d696efc16ffdbf9003f2fe8dd3630ccac4003ac066c32746d0: Status 404 returned error can't find the container with id 8e8dd47f6c74e7d696efc16ffdbf9003f2fe8dd3630ccac4003ac066c32746d0 Jul 28 15:40:53.077284 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:53.077250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66755c677b-tst75" event={"ID":"dbabdc75-6227-492f-aede-af87d3ef8158","Type":"ContainerStarted","Data":"d128cb4ec161a8977e1be3f0371d3fb0863cb28b76066bbeedf0e5910b808979"} Jul 28 15:40:53.077479 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:53.077302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66755c677b-tst75" event={"ID":"dbabdc75-6227-492f-aede-af87d3ef8158","Type":"ContainerStarted","Data":"8e8dd47f6c74e7d696efc16ffdbf9003f2fe8dd3630ccac4003ac066c32746d0"} Jul 28 15:40:53.099819 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:40:53.099766 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66755c677b-tst75" podStartSLOduration=1.099748957 podStartE2EDuration="1.099748957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:53.098232743 +0000 UTC m=+471.299080234" watchObservedRunningTime="2026-07-28 15:40:53.099748957 +0000 UTC m=+471.300596448" Jul 28 15:41:01.071758 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:01.071726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-zvg49" Jul 28 15:41:02.560899 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:02.560859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:41:02.561286 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:02.560938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:41:02.565901 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:02.565879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:41:03.118265 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:03.118234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66755c677b-tst75" Jul 28 15:41:03.178009 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:03.177974 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:41:28.197282 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.197238 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6659746ccd-tcskk" podUID="b334e833-bfd5-43f4-becd-1b2db5941425" containerName="console" containerID="cri-o://f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01" gracePeriod=15 Jul 28 15:41:28.439487 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.439462 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:41:28.620661 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620630 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.620821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620684 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.620821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620721 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.620821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.620821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620795 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjbxf\" (UniqueName: \"kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.620821 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.621007 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.620864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert\") pod \"b334e833-bfd5-43f4-becd-1b2db5941425\" (UID: \"b334e833-bfd5-43f4-becd-1b2db5941425\") " Jul 28 15:41:28.621303 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.621172 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config" (OuterVolumeSpecName: "console-config") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:28.621440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.621132 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:28.621440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.621303 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca" (OuterVolumeSpecName: "service-ca") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:28.621440 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.621324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:28.622951 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.622921 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:41:28.623054 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.622994 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:41:28.623107 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.623048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf" (OuterVolumeSpecName: "kube-api-access-pjbxf") pod "b334e833-bfd5-43f4-becd-1b2db5941425" (UID: "b334e833-bfd5-43f4-becd-1b2db5941425"). InnerVolumeSpecName "kube-api-access-pjbxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:41:28.721782 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721751 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.721782 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721776 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbxf\" (UniqueName: \"kubernetes.io/projected/b334e833-bfd5-43f4-becd-1b2db5941425-kube-api-access-pjbxf\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.721782 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721786 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-service-ca\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.722000 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721804 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-oauth-serving-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.722000 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721812 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-trusted-ca-bundle\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.722000 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721820 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b334e833-bfd5-43f4-becd-1b2db5941425-console-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:28.722000 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:28.721829 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b334e833-bfd5-43f4-becd-1b2db5941425-console-oauth-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:41:29.207285 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.207245 2575 generic.go:356] "Generic (PLEG): container finished" podID="b334e833-bfd5-43f4-becd-1b2db5941425" containerID="f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01" exitCode=0 Jul 28 15:41:29.207723 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.207292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6659746ccd-tcskk" Jul 28 15:41:29.207723 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.207292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6659746ccd-tcskk" event={"ID":"b334e833-bfd5-43f4-becd-1b2db5941425","Type":"ContainerDied","Data":"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01"} Jul 28 15:41:29.207723 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.207434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6659746ccd-tcskk" event={"ID":"b334e833-bfd5-43f4-becd-1b2db5941425","Type":"ContainerDied","Data":"4f5456e34c5bc32707f1fe3452c479380d4f1519df9715c5ff4f7bfb950d3a28"} Jul 28 15:41:29.207723 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.207467 2575 scope.go:117] "RemoveContainer" containerID="f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01" Jul 28 15:41:29.216267 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.216250 2575 scope.go:117] "RemoveContainer" containerID="f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01" Jul 28 15:41:29.216586 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:41:29.216560 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01\": container with ID starting with f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01 not found: ID does not exist" containerID="f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01" Jul 28 15:41:29.216673 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.216588 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01"} err="failed to get container status \"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01\": rpc error: code = NotFound desc = could not find container \"f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01\": container with ID starting with f29e38bce5a09aec40d23204dee3fdc507a550df34cb459855866fe7d4853d01 not found: ID does not exist" Jul 28 15:41:29.227994 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.227967 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:41:29.231539 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:29.231514 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6659746ccd-tcskk"] Jul 28 15:41:30.373613 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:41:30.373580 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b334e833-bfd5-43f4-becd-1b2db5941425" path="/var/lib/kubelet/pods/b334e833-bfd5-43f4-becd-1b2db5941425/volumes" Jul 28 15:42:22.334306 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.334272 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:22.334731 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.334570 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b334e833-bfd5-43f4-becd-1b2db5941425" containerName="console" Jul 28 15:42:22.334731 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.334583 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b334e833-bfd5-43f4-becd-1b2db5941425" containerName="console" Jul 28 15:42:22.334731 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.334621 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b334e833-bfd5-43f4-becd-1b2db5941425" containerName="console" Jul 28 15:42:22.338524 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.338503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.340617 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.340593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Jul 28 15:42:22.340704 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.340637 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7l4qw\"" Jul 28 15:42:22.361760 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.361727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:22.416297 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.416257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.416507 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.416331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgv5\" (UniqueName: \"kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.517588 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.517552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.517747 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.517635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgv5\" (UniqueName: \"kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.517972 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.517947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.526589 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.526556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgv5\" (UniqueName: \"kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5\") pod \"seaweedfs-tls-custom-6644bf5d6c-2c6rr\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.647007 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.646921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:22.772143 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:22.772113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:22.775215 ip-10-0-136-8 kubenswrapper[2575]: W0728 15:42:22.775185 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae690c60_7134_4199_923b_e0d415a4bb15.slice/crio-87d3bf35f2107cd50b0e5f483c7a0c844be4df71e0784ba157ed134718a5f705 WatchSource:0}: Error finding container 87d3bf35f2107cd50b0e5f483c7a0c844be4df71e0784ba157ed134718a5f705: Status 404 returned error can't find the container with id 87d3bf35f2107cd50b0e5f483c7a0c844be4df71e0784ba157ed134718a5f705 Jul 28 15:42:23.390068 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:23.390003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" event={"ID":"ae690c60-7134-4199-923b-e0d415a4bb15","Type":"ContainerStarted","Data":"87d3bf35f2107cd50b0e5f483c7a0c844be4df71e0784ba157ed134718a5f705"} Jul 28 15:42:26.407047 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:26.407001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" event={"ID":"ae690c60-7134-4199-923b-e0d415a4bb15","Type":"ContainerStarted","Data":"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f"} Jul 28 15:42:26.422860 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:26.422807 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" podStartSLOduration=1.260292465 podStartE2EDuration="4.422792239s" totalImagesPullingTime="3.162499774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:42:26.421854848 +0000 UTC m=+564.622702339" watchObservedRunningTime="2026-07-28 15:42:26.422792239 +0000 UTC m=+564.623639707" Jul 28 15:42:28.658760 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:28.658724 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:28.659210 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:28.658904 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" podUID="ae690c60-7134-4199-923b-e0d415a4bb15" containerName="seaweedfs-tls-custom" containerID="cri-o://54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f" gracePeriod=30 Jul 28 15:42:29.893000 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:29.892976 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:29.981231 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:29.981140 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data\") pod \"ae690c60-7134-4199-923b-e0d415a4bb15\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " Jul 28 15:42:29.981231 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:29.981215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmgv5\" (UniqueName: \"kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5\") pod \"ae690c60-7134-4199-923b-e0d415a4bb15\" (UID: \"ae690c60-7134-4199-923b-e0d415a4bb15\") " Jul 28 15:42:29.982374 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:29.982348 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data" (OuterVolumeSpecName: "data") pod "ae690c60-7134-4199-923b-e0d415a4bb15" (UID: "ae690c60-7134-4199-923b-e0d415a4bb15"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:42:29.983348 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:29.983324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5" (OuterVolumeSpecName: "kube-api-access-fmgv5") pod "ae690c60-7134-4199-923b-e0d415a4bb15" (UID: "ae690c60-7134-4199-923b-e0d415a4bb15"). InnerVolumeSpecName "kube-api-access-fmgv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:42:30.081954 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.081913 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmgv5\" (UniqueName: \"kubernetes.io/projected/ae690c60-7134-4199-923b-e0d415a4bb15-kube-api-access-fmgv5\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:42:30.081954 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.081948 2575 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae690c60-7134-4199-923b-e0d415a4bb15-data\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 15:42:30.422307 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.422273 2575 generic.go:356] "Generic (PLEG): container finished" podID="ae690c60-7134-4199-923b-e0d415a4bb15" containerID="54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f" exitCode=0 Jul 28 15:42:30.422467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.422323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" Jul 28 15:42:30.422467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.422349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" event={"ID":"ae690c60-7134-4199-923b-e0d415a4bb15","Type":"ContainerDied","Data":"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f"} Jul 28 15:42:30.422467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.422413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr" event={"ID":"ae690c60-7134-4199-923b-e0d415a4bb15","Type":"ContainerDied","Data":"87d3bf35f2107cd50b0e5f483c7a0c844be4df71e0784ba157ed134718a5f705"} Jul 28 15:42:30.422467 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.422436 2575 scope.go:117] "RemoveContainer" containerID="54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f" Jul 28 15:42:30.431754 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.431732 2575 scope.go:117] "RemoveContainer" containerID="54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f" Jul 28 15:42:30.432039 ip-10-0-136-8 kubenswrapper[2575]: E0728 15:42:30.432020 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f\": container with ID starting with 54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f not found: ID does not exist" containerID="54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f" Jul 28 15:42:30.432107 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.432048 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f"} err="failed to get container status \"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f\": rpc error: code = NotFound desc = could not find container \"54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f\": container with ID starting with 54f936d20546063f8302b24bc1d0448bf2ce3ff1fb1e1b1bd4ede9c4d0dfc28f not found: ID does not exist" Jul 28 15:42:30.437786 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.437762 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:30.442154 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:30.442133 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2c6rr"] Jul 28 15:42:32.372121 ip-10-0-136-8 kubenswrapper[2575]: I0728 15:42:32.372087 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae690c60-7134-4199-923b-e0d415a4bb15" path="/var/lib/kubelet/pods/ae690c60-7134-4199-923b-e0d415a4bb15/volumes" Jul 28 16:02:41.946117 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.946076 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:02:41.946571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.946332 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae690c60-7134-4199-923b-e0d415a4bb15" containerName="seaweedfs-tls-custom" Jul 28 16:02:41.946571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.946345 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae690c60-7134-4199-923b-e0d415a4bb15" containerName="seaweedfs-tls-custom" Jul 28 16:02:41.946571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.946400 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae690c60-7134-4199-923b-e0d415a4bb15" containerName="seaweedfs-tls-custom" Jul 28 16:02:41.948547 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.948527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:41.950604 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.950578 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Jul 28 16:02:41.950844 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.950829 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:02:41.950923 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.950891 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:02:41.950923 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.950899 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:02:41.951177 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.951157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:02:41.951268 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.951204 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Jul 28 16:02:41.951334 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.951264 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:02:41.958980 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:41.958959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:02:42.009175 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.009139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.009362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.009194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.009362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.009236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.009362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.009256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.009362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.009272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj87k\" (UniqueName: \"kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.109668 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.109638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.109827 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.109676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.109827 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.109701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.109827 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.109718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj87k\" (UniqueName: \"kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.109827 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.109748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.110105 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.110083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.110416 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.110370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.110506 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.110483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.112107 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.112087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.118679 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.118656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj87k\" (UniqueName: \"kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5n7rq\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.258412 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.258361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:02:42.389087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.389058 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:02:42.392130 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:02:42.392105 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3cad72_4ae4_49df_aebc_044f32f078cb.slice/crio-25e876ac6e310feb9cee27936cf0b817c6eaa067c10e3eff661f81e5d96da3da WatchSource:0}: Error finding container 25e876ac6e310feb9cee27936cf0b817c6eaa067c10e3eff661f81e5d96da3da: Status 404 returned error can't find the container with id 25e876ac6e310feb9cee27936cf0b817c6eaa067c10e3eff661f81e5d96da3da Jul 28 16:02:42.394126 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.394111 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 16:02:42.553744 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:42.553656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerStarted","Data":"25e876ac6e310feb9cee27936cf0b817c6eaa067c10e3eff661f81e5d96da3da"} Jul 28 16:02:47.577629 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:47.577586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerStarted","Data":"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947"} Jul 28 16:02:50.589481 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:50.589444 2575 generic.go:356] "Generic (PLEG): container finished" podID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerID="5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947" exitCode=0 Jul 28 16:02:50.589885 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:02:50.589518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerDied","Data":"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947"} Jul 28 16:03:04.652888 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:04.652849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerStarted","Data":"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9"} Jul 28 16:03:06.661744 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:06.661711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerStarted","Data":"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b"} Jul 28 16:03:06.662200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:06.661908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:03:06.662200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:06.662049 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:03:06.663244 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:06.663198 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:06.684650 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:06.684602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podStartSLOduration=2.097781689 podStartE2EDuration="25.684588614s" totalImagesPullingTime="19.85576881s" totalInitContainerRuntime="3.731038115s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:06.683686988 +0000 UTC m=+1804.884534478" watchObservedRunningTime="2026-07-28 16:03:06.684588614 +0000 UTC m=+1804.885436145" Jul 28 16:03:07.665300 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:07.665254 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:12.669692 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:12.669662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:03:12.670263 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:12.670236 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:22.670748 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:22.670710 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:32.670313 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:32.670272 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:42.670226 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:42.670187 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:03:52.670495 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:03:52.670456 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:04:02.670275 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:02.670234 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:04:12.670975 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:12.670932 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:04:22.670599 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:22.670565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:04:32.202482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.202448 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:04:32.202940 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.202681 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" containerID="cri-o://bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9" gracePeriod=30 Jul 28 16:04:32.202940 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.202731 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kube-rbac-proxy" containerID="cri-o://74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b" gracePeriod=30 Jul 28 16:04:32.666482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.666446 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused" Jul 28 16:04:32.670780 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.670753 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" Jul 28 16:04:32.954691 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.954596 2575 generic.go:356] "Generic (PLEG): container finished" podID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerID="74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b" exitCode=2 Jul 28 16:04:32.954691 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:32.954652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerDied","Data":"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b"} Jul 28 16:04:36.953836 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.953808 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:04:36.970026 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.969991 2575 generic.go:356] "Generic (PLEG): container finished" podID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerID="bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9" exitCode=0 Jul 28 16:04:36.970154 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.970049 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" Jul 28 16:04:36.970154 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.970068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerDied","Data":"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9"} Jul 28 16:04:36.970154 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.970132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq" event={"ID":"7f3cad72-4ae4-49df-aebc-044f32f078cb","Type":"ContainerDied","Data":"25e876ac6e310feb9cee27936cf0b817c6eaa067c10e3eff661f81e5d96da3da"} Jul 28 16:04:36.970254 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.970159 2575 scope.go:117] "RemoveContainer" containerID="74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b" Jul 28 16:04:36.978627 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.978603 2575 scope.go:117] "RemoveContainer" containerID="bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9" Jul 28 16:04:36.986729 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.986707 2575 scope.go:117] "RemoveContainer" containerID="5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947" Jul 28 16:04:36.993902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.993881 2575 scope.go:117] "RemoveContainer" containerID="74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b" Jul 28 16:04:36.994166 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:04:36.994148 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b\": container with ID starting with 74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b not found: ID does not exist" containerID="74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b" Jul 28 16:04:36.994213 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.994176 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b"} err="failed to get container status \"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b\": rpc error: code = NotFound desc = could not find container \"74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b\": container with ID starting with 74f4ff140b685fe811c61c3aff21d566845ee1bf399ca500ec95afbb09477d4b not found: ID does not exist" Jul 28 16:04:36.994213 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.994194 2575 scope.go:117] "RemoveContainer" containerID="bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9" Jul 28 16:04:36.994439 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:04:36.994420 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9\": container with ID starting with bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9 not found: ID does not exist" containerID="bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9" Jul 28 16:04:36.994499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.994445 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9"} err="failed to get container status \"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9\": rpc error: code = NotFound desc = could not find container \"bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9\": container with ID starting with bf8646453eba165ec78ac4d6cf0c159ee9e70a31c1ca13a6825d0d99273a66e9 not found: ID does not exist" Jul 28 16:04:36.994499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.994462 2575 scope.go:117] "RemoveContainer" containerID="5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947" Jul 28 16:04:36.994677 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:04:36.994663 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947\": container with ID starting with 5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947 not found: ID does not exist" containerID="5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947" Jul 28 16:04:36.994715 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:36.994680 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947"} err="failed to get container status \"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947\": rpc error: code = NotFound desc = could not find container \"5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947\": container with ID starting with 5f4478e1c8f987d44db9b195bb92a2241e7c51a99a3586f233fbde16f2f28947 not found: ID does not exist" Jul 28 16:04:37.082220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082127 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj87k\" (UniqueName: \"kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k\") pod \"7f3cad72-4ae4-49df-aebc-044f32f078cb\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " Jul 28 16:04:37.082220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082171 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert\") pod \"7f3cad72-4ae4-49df-aebc-044f32f078cb\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " Jul 28 16:04:37.082220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082226 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls\") pod \"7f3cad72-4ae4-49df-aebc-044f32f078cb\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " Jul 28 16:04:37.082596 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082295 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location\") pod \"7f3cad72-4ae4-49df-aebc-044f32f078cb\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " Jul 28 16:04:37.082596 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config\") pod \"7f3cad72-4ae4-49df-aebc-044f32f078cb\" (UID: \"7f3cad72-4ae4-49df-aebc-044f32f078cb\") " Jul 28 16:04:37.082691 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082644 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f3cad72-4ae4-49df-aebc-044f32f078cb" (UID: "7f3cad72-4ae4-49df-aebc-044f32f078cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:04:37.082732 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082685 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f3cad72-4ae4-49df-aebc-044f32f078cb" (UID: "7f3cad72-4ae4-49df-aebc-044f32f078cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:04:37.082732 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.082702 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f3cad72-4ae4-49df-aebc-044f32f078cb" (UID: "7f3cad72-4ae4-49df-aebc-044f32f078cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:04:37.084446 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.084420 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f3cad72-4ae4-49df-aebc-044f32f078cb" (UID: "7f3cad72-4ae4-49df-aebc-044f32f078cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:04:37.084546 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.084453 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k" (OuterVolumeSpecName: "kube-api-access-sj87k") pod "7f3cad72-4ae4-49df-aebc-044f32f078cb" (UID: "7f3cad72-4ae4-49df-aebc-044f32f078cb"). InnerVolumeSpecName "kube-api-access-sj87k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:04:37.183805 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.183769 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3cad72-4ae4-49df-aebc-044f32f078cb-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:04:37.183805 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.183798 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:04:37.183805 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.183807 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj87k\" (UniqueName: \"kubernetes.io/projected/7f3cad72-4ae4-49df-aebc-044f32f078cb-kube-api-access-sj87k\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:04:37.184029 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.183818 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3cad72-4ae4-49df-aebc-044f32f078cb-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:04:37.184029 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.183826 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f3cad72-4ae4-49df-aebc-044f32f078cb-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:04:37.293454 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.293424 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:04:37.298553 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.298528 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5n7rq"] Jul 28 16:04:37.314187 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314154 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:04:37.314433 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314420 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="storage-initializer" Jul 28 16:04:37.314482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314435 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="storage-initializer" Jul 28 16:04:37.314482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314443 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" Jul 28 16:04:37.314482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314448 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" Jul 28 16:04:37.314482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314455 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kube-rbac-proxy" Jul 28 16:04:37.314482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314460 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kube-rbac-proxy" Jul 28 16:04:37.314628 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314513 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kserve-container" Jul 28 16:04:37.314628 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.314523 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" containerName="kube-rbac-proxy" Jul 28 16:04:37.318008 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.317991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.319926 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.319903 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Jul 28 16:04:37.320041 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.319954 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Jul 28 16:04:37.320207 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.320188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:04:37.320353 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.320336 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:04:37.320601 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.320559 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:04:37.320745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.320604 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:04:37.320745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.320625 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:04:37.329079 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.329057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:04:37.386278 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.386186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.386278 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.386225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5kb\" (UniqueName: \"kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.386278 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.386250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.386536 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.386341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.386536 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.386385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487345 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5kb\" (UniqueName: \"kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.487965 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.487936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.488308 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.488287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.488414 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.488375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.490057 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.490038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.495057 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.495036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5kb\" (UniqueName: \"kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.629022 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.628982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:37.756814 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.756781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:04:37.759786 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:04:37.759759 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161c8d9f_19c3_4c54_b0f4_4ba7f748b35d.slice/crio-bf4e9a2825923efbe576a1e0ebe1c75aa3ab0d65f1dd927192efb1121f441a6e WatchSource:0}: Error finding container bf4e9a2825923efbe576a1e0ebe1c75aa3ab0d65f1dd927192efb1121f441a6e: Status 404 returned error can't find the container with id bf4e9a2825923efbe576a1e0ebe1c75aa3ab0d65f1dd927192efb1121f441a6e Jul 28 16:04:37.975439 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.975325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerStarted","Data":"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593"} Jul 28 16:04:37.975439 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:37.975413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerStarted","Data":"bf4e9a2825923efbe576a1e0ebe1c75aa3ab0d65f1dd927192efb1121f441a6e"} Jul 28 16:04:38.372549 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:38.372509 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3cad72-4ae4-49df-aebc-044f32f078cb" path="/var/lib/kubelet/pods/7f3cad72-4ae4-49df-aebc-044f32f078cb/volumes" Jul 28 16:04:41.990808 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:41.990774 2575 generic.go:356] "Generic (PLEG): container finished" podID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerID="b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593" exitCode=0 Jul 28 16:04:41.991179 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:41.990821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerDied","Data":"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593"} Jul 28 16:04:42.996405 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:42.996360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerStarted","Data":"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b"} Jul 28 16:04:42.996863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:42.996472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerStarted","Data":"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b"} Jul 28 16:04:42.996863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:42.996795 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:43.020103 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:43.020049 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podStartSLOduration=2.085113785 podStartE2EDuration="6.020033916s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934920131s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:43.019544584 +0000 UTC m=+1901.220392075" watchObservedRunningTime="2026-07-28 16:04:43.020033916 +0000 UTC m=+1901.220881407" Jul 28 16:04:44.002862 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:44.002799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:44.004234 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:44.004199 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:04:45.006682 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:45.006632 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:04:50.011086 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:50.011054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:04:50.011658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:04:50.011632 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:00.011751 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:00.011702 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:10.012200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:10.012153 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:20.012067 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:20.012020 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:30.012203 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:30.012157 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:40.012467 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:40.012425 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:05:50.012001 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:05:50.011958 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:06:00.013241 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:00.013161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:06:07.343939 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.343899 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:06:07.344453 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.344269 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" containerID="cri-o://1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b" gracePeriod=30 Jul 28 16:06:07.344453 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.344314 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kube-rbac-proxy" containerID="cri-o://816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b" gracePeriod=30 Jul 28 16:06:07.423686 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.423654 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:06:07.426093 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.426076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.428009 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.427991 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Jul 28 16:06:07.439515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.439492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:06:07.476435 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.476385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fb5\" (UniqueName: \"kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.476634 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.476463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.476634 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.476499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.476634 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.476518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.476634 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.476606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577362 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fb5\" (UniqueName: \"kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577588 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577588 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577588 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.577786 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.577762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.578107 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.578084 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.578267 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.578177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.579866 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.579849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.586136 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.586117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fb5\" (UniqueName: \"kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.736019 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.735940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:07.864150 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:07.864126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:06:07.866688 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:06:07.866664 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4fe046_6e1f_434b_ac6c_5acf25b2ec36.slice/crio-6a49ce9ee5ecf9c19947612c34307e79181209974d9906c46e1bbf1be241091b WatchSource:0}: Error finding container 6a49ce9ee5ecf9c19947612c34307e79181209974d9906c46e1bbf1be241091b: Status 404 returned error can't find the container with id 6a49ce9ee5ecf9c19947612c34307e79181209974d9906c46e1bbf1be241091b Jul 28 16:06:08.300071 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:08.300035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerStarted","Data":"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb"} Jul 28 16:06:08.300277 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:08.300111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerStarted","Data":"6a49ce9ee5ecf9c19947612c34307e79181209974d9906c46e1bbf1be241091b"} Jul 28 16:06:08.302373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:08.302346 2575 generic.go:356] "Generic (PLEG): container finished" podID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerID="816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b" exitCode=2 Jul 28 16:06:08.302507 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:08.302418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerDied","Data":"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b"} Jul 28 16:06:10.007458 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:10.007422 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused" Jul 28 16:06:10.012228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:10.012200 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 28 16:06:12.293531 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.293506 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:06:12.317861 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.317824 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerID="e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb" exitCode=0 Jul 28 16:06:12.318013 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.317906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerDied","Data":"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb"} Jul 28 16:06:12.319903 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.319875 2575 generic.go:356] "Generic (PLEG): container finished" podID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerID="1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b" exitCode=0 Jul 28 16:06:12.320012 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.319938 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" Jul 28 16:06:12.320012 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.319957 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerDied","Data":"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b"} Jul 28 16:06:12.320102 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.320010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt" event={"ID":"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d","Type":"ContainerDied","Data":"bf4e9a2825923efbe576a1e0ebe1c75aa3ab0d65f1dd927192efb1121f441a6e"} Jul 28 16:06:12.320102 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.320050 2575 scope.go:117] "RemoveContainer" containerID="816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b" Jul 28 16:06:12.329812 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.329793 2575 scope.go:117] "RemoveContainer" containerID="1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b" Jul 28 16:06:12.343251 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.343232 2575 scope.go:117] "RemoveContainer" containerID="b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593" Jul 28 16:06:12.357980 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.357954 2575 scope.go:117] "RemoveContainer" containerID="816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b" Jul 28 16:06:12.358339 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:06:12.358316 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b\": container with ID starting with 816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b not found: ID does not exist" containerID="816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b" Jul 28 16:06:12.358424 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.358351 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b"} err="failed to get container status \"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b\": rpc error: code = NotFound desc = could not find container \"816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b\": container with ID starting with 816f6e9bd09422348a5f8cb2c05ca2912acaf10feb1c816bc5c6e56ed1f97f8b not found: ID does not exist" Jul 28 16:06:12.358424 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.358371 2575 scope.go:117] "RemoveContainer" containerID="1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b" Jul 28 16:06:12.358658 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:06:12.358639 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b\": container with ID starting with 1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b not found: ID does not exist" containerID="1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b" Jul 28 16:06:12.358753 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.358663 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b"} err="failed to get container status \"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b\": rpc error: code = NotFound desc = could not find container \"1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b\": container with ID starting with 1a61c01e3335f20b8511f9a79b18bc3e5b494531d6e46d7ad8fa520452558c8b not found: ID does not exist" Jul 28 16:06:12.358753 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.358679 2575 scope.go:117] "RemoveContainer" containerID="b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593" Jul 28 16:06:12.358964 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:06:12.358937 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593\": container with ID starting with b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593 not found: ID does not exist" containerID="b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593" Jul 28 16:06:12.359036 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.358971 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593"} err="failed to get container status \"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593\": rpc error: code = NotFound desc = could not find container \"b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593\": container with ID starting with b1d8dc22ded0cd04e2666953d61e1a0e47df0a84178e196388e61441f49fb593 not found: ID does not exist" Jul 28 16:06:12.416600 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416577 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config\") pod \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " Jul 28 16:06:12.416719 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416613 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location\") pod \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " Jul 28 16:06:12.416719 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416643 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls\") pod \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " Jul 28 16:06:12.416719 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416688 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert\") pod \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " Jul 28 16:06:12.416815 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416729 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d5kb\" (UniqueName: \"kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb\") pod \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\" (UID: \"161c8d9f-19c3-4c54-b0f4-4ba7f748b35d\") " Jul 28 16:06:12.416951 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" (UID: "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:06:12.417007 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.416961 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" (UID: "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:06:12.417064 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.417041 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" (UID: "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:06:12.417157 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.417134 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:06:12.417210 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.417163 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:06:12.417210 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.417177 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:06:12.418850 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.418831 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" (UID: "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:06:12.418922 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.418852 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb" (OuterVolumeSpecName: "kube-api-access-6d5kb") pod "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" (UID: "161c8d9f-19c3-4c54-b0f4-4ba7f748b35d"). InnerVolumeSpecName "kube-api-access-6d5kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:06:12.517839 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.517801 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:06:12.517839 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.517834 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5kb\" (UniqueName: \"kubernetes.io/projected/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d-kube-api-access-6d5kb\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:06:12.648248 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.648209 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:06:12.652252 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:12.652226 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-ntjlt"] Jul 28 16:06:13.326124 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:13.326086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerStarted","Data":"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056"} Jul 28 16:06:13.326599 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:13.326196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerStarted","Data":"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6"} Jul 28 16:06:13.326599 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:13.326427 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:13.348807 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:13.348757 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podStartSLOduration=2.657585086 podStartE2EDuration="6.348742285s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.691157199s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:06:13.347975484 +0000 UTC m=+1991.548822976" watchObservedRunningTime="2026-07-28 16:06:13.348742285 +0000 UTC m=+1991.549589775" Jul 28 16:06:14.330423 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:14.330380 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:14.331661 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:14.331626 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:06:14.373523 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:14.373487 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" path="/var/lib/kubelet/pods/161c8d9f-19c3-4c54-b0f4-4ba7f748b35d/volumes" Jul 28 16:06:15.334437 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:15.334402 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:06:20.339451 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:20.339420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:06:20.340067 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:20.340034 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:06:30.340415 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:30.340354 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:06:40.339975 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:40.339930 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:06:50.340455 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:06:50.340416 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:07:00.340352 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:00.340308 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:07:10.340663 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:10.340621 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:07:20.340554 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:20.340515 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:07:30.340598 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:30.340516 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:07:37.558976 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:37.558946 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:07:37.559538 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:37.559205 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" containerID="cri-o://36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6" gracePeriod=30 Jul 28 16:07:37.559538 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:37.559246 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kube-rbac-proxy" containerID="cri-o://3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056" gracePeriod=30 Jul 28 16:07:38.620213 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:38.620176 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerID="3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056" exitCode=2 Jul 28 16:07:38.620608 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:38.620248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerDied","Data":"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056"} Jul 28 16:07:40.335590 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:40.335544 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused" Jul 28 16:07:40.339903 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:40.339873 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 28 16:07:42.504887 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.504856 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:07:42.617545 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617461 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location\") pod \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " Jul 28 16:07:42.617545 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4fb5\" (UniqueName: \"kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5\") pod \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " Jul 28 16:07:42.617545 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617545 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert\") pod \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " Jul 28 16:07:42.617796 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config\") pod \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " Jul 28 16:07:42.617796 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617603 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls\") pod \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\" (UID: \"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36\") " Jul 28 16:07:42.617900 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617842 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" (UID: "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:07:42.617957 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617929 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" (UID: "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:07:42.618016 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.617977 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" (UID: "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:07:42.619781 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.619754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" (UID: "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:07:42.619864 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.619823 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5" (OuterVolumeSpecName: "kube-api-access-x4fb5") pod "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" (UID: "7c4fe046-6e1f-434b-ac6c-5acf25b2ec36"). InnerVolumeSpecName "kube-api-access-x4fb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:07:42.635783 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.635749 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerID="36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6" exitCode=0 Jul 28 16:07:42.635927 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.635813 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" Jul 28 16:07:42.635927 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.635834 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerDied","Data":"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6"} Jul 28 16:07:42.635927 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.635921 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs" event={"ID":"7c4fe046-6e1f-434b-ac6c-5acf25b2ec36","Type":"ContainerDied","Data":"6a49ce9ee5ecf9c19947612c34307e79181209974d9906c46e1bbf1be241091b"} Jul 28 16:07:42.636028 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.635950 2575 scope.go:117] "RemoveContainer" containerID="3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056" Jul 28 16:07:42.644022 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.644005 2575 scope.go:117] "RemoveContainer" containerID="36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6" Jul 28 16:07:42.651807 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.651789 2575 scope.go:117] "RemoveContainer" containerID="e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb" Jul 28 16:07:42.659672 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.659606 2575 scope.go:117] "RemoveContainer" containerID="3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056" Jul 28 16:07:42.659948 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:07:42.659928 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056\": container with ID starting with 3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056 not found: ID does not exist" containerID="3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056" Jul 28 16:07:42.659986 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.659959 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056"} err="failed to get container status \"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056\": rpc error: code = NotFound desc = could not find container \"3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056\": container with ID starting with 3efcbb82266edf6b53c90be6df81684ebb477c98522a5fdd0fa0f02b36af3056 not found: ID does not exist" Jul 28 16:07:42.659986 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.659979 2575 scope.go:117] "RemoveContainer" containerID="36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6" Jul 28 16:07:42.660247 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:07:42.660230 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6\": container with ID starting with 36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6 not found: ID does not exist" containerID="36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6" Jul 28 16:07:42.660283 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.660254 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6"} err="failed to get container status \"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6\": rpc error: code = NotFound desc = could not find container \"36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6\": container with ID starting with 36d49243c4bf374e195ce6da0b9d878affbb18f393bdac8b62853f429c3cf4c6 not found: ID does not exist" Jul 28 16:07:42.660283 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.660270 2575 scope.go:117] "RemoveContainer" containerID="e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb" Jul 28 16:07:42.660595 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:07:42.660576 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb\": container with ID starting with e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb not found: ID does not exist" containerID="e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb" Jul 28 16:07:42.660650 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.660601 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb"} err="failed to get container status \"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb\": rpc error: code = NotFound desc = could not find container \"e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb\": container with ID starting with e3063aee2803847a6fce0c08e548d16cc54a1b4c775fb3b69e571a3c3ecb49bb not found: ID does not exist" Jul 28 16:07:42.663042 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.663022 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:07:42.671087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.671049 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-tm7gs"] Jul 28 16:07:42.718242 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.718208 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:07:42.718482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.718465 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:07:42.718583 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.718570 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4fb5\" (UniqueName: \"kubernetes.io/projected/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-api-access-x4fb5\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:07:42.718673 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.718653 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:07:42.718673 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.718674 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:07:42.791984 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.791944 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:07:42.792302 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792288 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kube-rbac-proxy" Jul 28 16:07:42.792351 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792305 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kube-rbac-proxy" Jul 28 16:07:42.792351 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792320 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kube-rbac-proxy" Jul 28 16:07:42.792351 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792329 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kube-rbac-proxy" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792352 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="storage-initializer" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792363 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="storage-initializer" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792381 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792406 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792414 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792427 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="storage-initializer" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792435 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="storage-initializer" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792494 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kube-rbac-proxy" Jul 28 16:07:42.792510 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792504 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kube-rbac-proxy" Jul 28 16:07:42.792801 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792515 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="161c8d9f-19c3-4c54-b0f4-4ba7f748b35d" containerName="kserve-container" Jul 28 16:07:42.792801 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.792523 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" containerName="kserve-container" Jul 28 16:07:42.797309 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.797285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:42.799669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.799641 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Jul 28 16:07:42.799826 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.799705 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:07:42.799915 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.799818 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:07:42.800089 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.800072 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:07:42.800179 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.800158 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:07:42.800250 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.800164 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Jul 28 16:07:42.800250 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.800210 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:07:42.811367 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.811343 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:07:42.920385 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.920296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:42.920385 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.920345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz7d\" (UniqueName: \"kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:42.920604 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.920438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:42.920604 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.920476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:42.920604 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:42.920514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021184 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz7d\" (UniqueName: \"kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:07:43.021321 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021499 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:07:43.021434 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls podName:07a5624c-34a7-4d35-8090-5b187cc22e53 nodeName:}" failed. No retries permitted until 2026-07-28 16:07:43.521411074 +0000 UTC m=+2081.722258558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-gpr6d" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53") : secret "isvc-canary-stop-predictor-serving-cert" not found Jul 28 16:07:43.021830 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.021997 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.021978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.022051 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.022032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.032095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.032068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz7d\" (UniqueName: \"kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.525601 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.525555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.528053 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.528031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-gpr6d\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.708738 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.708701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:43.839489 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:43.839449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:07:43.842321 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:07:43.842293 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a5624c_34a7_4d35_8090_5b187cc22e53.slice/crio-07e46fa31d172a18c02672d23e9d7948c0051bbfd79c020bf6456ef2f29517b4 WatchSource:0}: Error finding container 07e46fa31d172a18c02672d23e9d7948c0051bbfd79c020bf6456ef2f29517b4: Status 404 returned error can't find the container with id 07e46fa31d172a18c02672d23e9d7948c0051bbfd79c020bf6456ef2f29517b4 Jul 28 16:07:44.373083 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:44.373050 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4fe046-6e1f-434b-ac6c-5acf25b2ec36" path="/var/lib/kubelet/pods/7c4fe046-6e1f-434b-ac6c-5acf25b2ec36/volumes" Jul 28 16:07:44.646231 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:44.646139 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerStarted","Data":"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7"} Jul 28 16:07:44.646231 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:44.646203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerStarted","Data":"07e46fa31d172a18c02672d23e9d7948c0051bbfd79c020bf6456ef2f29517b4"} Jul 28 16:07:48.663326 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:48.663291 2575 generic.go:356] "Generic (PLEG): container finished" podID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerID="0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7" exitCode=0 Jul 28 16:07:48.663735 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:48.663343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerDied","Data":"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7"} Jul 28 16:07:49.669239 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:49.669205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerStarted","Data":"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94"} Jul 28 16:07:49.669628 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:49.669268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerStarted","Data":"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9"} Jul 28 16:07:49.669628 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:49.669465 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:49.690523 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:49.690478 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podStartSLOduration=3.903960535 podStartE2EDuration="7.690463293s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786502758s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:49.689314199 +0000 UTC m=+2087.890161689" watchObservedRunningTime="2026-07-28 16:07:49.690463293 +0000 UTC m=+2087.891310853" Jul 28 16:07:50.672821 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:50.672786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:50.674047 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:50.674017 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:07:51.676037 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:51.675999 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:07:56.680602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:56.680566 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:07:56.681077 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:07:56.681038 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:06.681099 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:06.681053 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:16.681967 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:16.681928 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:26.680994 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:26.680956 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:36.681312 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:36.681273 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:46.681479 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:46.681441 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:08:56.681178 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:08:56.681087 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:09:06.682086 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:06.682057 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:09:12.771065 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:12.771029 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:09:12.771467 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:12.771279 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" containerID="cri-o://d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9" gracePeriod=30 Jul 28 16:09:12.771467 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:12.771310 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kube-rbac-proxy" containerID="cri-o://50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94" gracePeriod=30 Jul 28 16:09:12.962221 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:12.962191 2575 generic.go:356] "Generic (PLEG): container finished" podID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerID="50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94" exitCode=2 Jul 28 16:09:12.962383 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:12.962254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerDied","Data":"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94"} Jul 28 16:09:16.676996 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:16.676945 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused" Jul 28 16:09:16.681482 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:16.681449 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 28 16:09:17.612716 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.612688 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:09:17.737929 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.737847 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pz7d\" (UniqueName: \"kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d\") pod \"07a5624c-34a7-4d35-8090-5b187cc22e53\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " Jul 28 16:09:17.737929 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.737905 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config\") pod \"07a5624c-34a7-4d35-8090-5b187cc22e53\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.737939 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location\") pod \"07a5624c-34a7-4d35-8090-5b187cc22e53\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.737989 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert\") pod \"07a5624c-34a7-4d35-8090-5b187cc22e53\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.738014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") pod \"07a5624c-34a7-4d35-8090-5b187cc22e53\" (UID: \"07a5624c-34a7-4d35-8090-5b187cc22e53\") " Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.738299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07a5624c-34a7-4d35-8090-5b187cc22e53" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.738315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07a5624c-34a7-4d35-8090-5b187cc22e53" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:09:17.738431 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.738305 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07a5624c-34a7-4d35-8090-5b187cc22e53" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:09:17.740061 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.740035 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d" (OuterVolumeSpecName: "kube-api-access-8pz7d") pod "07a5624c-34a7-4d35-8090-5b187cc22e53" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53"). InnerVolumeSpecName "kube-api-access-8pz7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:09:17.740179 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.740117 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07a5624c-34a7-4d35-8090-5b187cc22e53" (UID: "07a5624c-34a7-4d35-8090-5b187cc22e53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:09:17.839560 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.839520 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz7d\" (UniqueName: \"kubernetes.io/projected/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-api-access-8pz7d\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:09:17.839560 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.839553 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:09:17.839560 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.839562 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07a5624c-34a7-4d35-8090-5b187cc22e53-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:09:17.839792 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.839573 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07a5624c-34a7-4d35-8090-5b187cc22e53-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:09:17.839792 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.839583 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a5624c-34a7-4d35-8090-5b187cc22e53-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:09:17.981253 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.981217 2575 generic.go:356] "Generic (PLEG): container finished" podID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerID="d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9" exitCode=0 Jul 28 16:09:17.981447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.981265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerDied","Data":"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9"} Jul 28 16:09:17.981447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.981272 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" Jul 28 16:09:17.981447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.981316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d" event={"ID":"07a5624c-34a7-4d35-8090-5b187cc22e53","Type":"ContainerDied","Data":"07e46fa31d172a18c02672d23e9d7948c0051bbfd79c020bf6456ef2f29517b4"} Jul 28 16:09:17.981447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.981342 2575 scope.go:117] "RemoveContainer" containerID="50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94" Jul 28 16:09:17.990494 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.990476 2575 scope.go:117] "RemoveContainer" containerID="d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9" Jul 28 16:09:17.997868 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:17.997852 2575 scope.go:117] "RemoveContainer" containerID="0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7" Jul 28 16:09:18.003867 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.003838 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:09:18.005968 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.005947 2575 scope.go:117] "RemoveContainer" containerID="50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94" Jul 28 16:09:18.006250 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:09:18.006230 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94\": container with ID starting with 50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94 not found: ID does not exist" containerID="50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94" Jul 28 16:09:18.006319 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.006264 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94"} err="failed to get container status \"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94\": rpc error: code = NotFound desc = could not find container \"50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94\": container with ID starting with 50221d57dae31166e9a4178cab530c9d9884ee56e1621a9e7fc11f18ab226b94 not found: ID does not exist" Jul 28 16:09:18.006319 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.006292 2575 scope.go:117] "RemoveContainer" containerID="d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9" Jul 28 16:09:18.006610 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:09:18.006587 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9\": container with ID starting with d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9 not found: ID does not exist" containerID="d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9" Jul 28 16:09:18.006652 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.006619 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9"} err="failed to get container status \"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9\": rpc error: code = NotFound desc = could not find container \"d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9\": container with ID starting with d525fc00d97a62e1367d9630de3c461db563891fe995ff140d0185e0796096e9 not found: ID does not exist" Jul 28 16:09:18.006652 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.006636 2575 scope.go:117] "RemoveContainer" containerID="0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7" Jul 28 16:09:18.006914 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:09:18.006894 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7\": container with ID starting with 0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7 not found: ID does not exist" containerID="0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7" Jul 28 16:09:18.006971 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.006919 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7"} err="failed to get container status \"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7\": rpc error: code = NotFound desc = could not find container \"0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7\": container with ID starting with 0503bd95a0a88977b488b6899a6dd25935099b73dd463031bd091fdc66b092c7 not found: ID does not exist" Jul 28 16:09:18.009317 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.009294 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-gpr6d"] Jul 28 16:09:18.372435 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:18.372378 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" path="/var/lib/kubelet/pods/07a5624c-34a7-4d35-8090-5b187cc22e53/volumes" Jul 28 16:09:23.087823 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.087787 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088049 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088060 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088069 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kube-rbac-proxy" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088075 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kube-rbac-proxy" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088086 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="storage-initializer" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088094 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="storage-initializer" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088143 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kube-rbac-proxy" Jul 28 16:09:23.088195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.088150 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a5624c-34a7-4d35-8090-5b187cc22e53" containerName="kserve-container" Jul 28 16:09:23.092625 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.092603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.094590 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.094568 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:09:23.094697 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.094633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:09:23.094697 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.094684 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\"" Jul 28 16:09:23.095331 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.095306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:09:23.095460 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.095306 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:09:23.095460 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.095354 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\"" Jul 28 16:09:23.095460 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.095309 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:09:23.101085 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.101066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:09:23.179052 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.179011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.179196 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.179065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.179196 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.179126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbbm\" (UniqueName: \"kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.179196 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.179179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.179320 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.179200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.279829 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.279786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.279952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.279849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.279952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.279886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.279952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.279914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbbm\" (UniqueName: \"kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.279952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.279932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.280296 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.280263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.280592 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.280575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.280644 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.280599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.282290 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.282271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.289283 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.289256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbbm\" (UniqueName: \"kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm\") pod \"isvc-lightgbm-predictor-54f85ddf88-9qmk9\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.403229 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.403133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:23.538427 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:23.538378 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:09:23.541441 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:09:23.541414 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c5edca_62f6_4190_b221_a8293cb267c1.slice/crio-37fdcaf20f36069c4edfe1e31986fdf2779499ec36d4fbff8c6d6f5de4cd6b5b WatchSource:0}: Error finding container 37fdcaf20f36069c4edfe1e31986fdf2779499ec36d4fbff8c6d6f5de4cd6b5b: Status 404 returned error can't find the container with id 37fdcaf20f36069c4edfe1e31986fdf2779499ec36d4fbff8c6d6f5de4cd6b5b Jul 28 16:09:24.012205 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:24.012152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerStarted","Data":"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252"} Jul 28 16:09:24.012422 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:24.012242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerStarted","Data":"37fdcaf20f36069c4edfe1e31986fdf2779499ec36d4fbff8c6d6f5de4cd6b5b"} Jul 28 16:09:28.026853 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:28.026828 2575 generic.go:356] "Generic (PLEG): container finished" podID="23c5edca-62f6-4190-b221-a8293cb267c1" containerID="afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252" exitCode=0 Jul 28 16:09:28.027329 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:28.026870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerDied","Data":"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252"} Jul 28 16:09:28.028193 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:28.028169 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 16:09:35.055801 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:35.055764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerStarted","Data":"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969"} Jul 28 16:09:35.056176 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:35.055832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerStarted","Data":"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60"} Jul 28 16:09:35.056176 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:35.056030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:35.075968 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:35.075910 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podStartSLOduration=2.104475803 podStartE2EDuration="12.075888953s" totalImagesPullingTime="6.295276987s" totalInitContainerRuntime="3.676136163s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:35.074899052 +0000 UTC m=+2193.275746543" watchObservedRunningTime="2026-07-28 16:09:35.075888953 +0000 UTC m=+2193.276736447" Jul 28 16:09:36.059276 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:36.059249 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:36.060660 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:36.060631 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:09:37.062949 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:37.062910 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:09:42.067038 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:42.067009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:09:42.067508 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:42.067480 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:09:52.067974 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:09:52.067936 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:02.068280 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:02.068242 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:12.068075 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:12.068034 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:22.067847 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:22.067802 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:32.068332 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:32.068250 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:42.067678 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:42.067640 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 28 16:10:52.067973 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:52.067944 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:10:53.285065 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:53.285031 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:10:53.285457 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:53.285339 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" containerID="cri-o://161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60" gracePeriod=30 Jul 28 16:10:53.285457 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:53.285350 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kube-rbac-proxy" containerID="cri-o://c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969" gracePeriod=30 Jul 28 16:10:54.342675 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:54.342641 2575 generic.go:356] "Generic (PLEG): container finished" podID="23c5edca-62f6-4190-b221-a8293cb267c1" containerID="c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969" exitCode=2 Jul 28 16:10:54.343058 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:54.342711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerDied","Data":"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969"} Jul 28 16:10:57.063614 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.063571 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused" Jul 28 16:10:57.826679 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.826655 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:10:57.932939 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.932841 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config\") pod \"23c5edca-62f6-4190-b221-a8293cb267c1\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " Jul 28 16:10:57.932939 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.932900 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls\") pod \"23c5edca-62f6-4190-b221-a8293cb267c1\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " Jul 28 16:10:57.932939 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.932927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert\") pod \"23c5edca-62f6-4190-b221-a8293cb267c1\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " Jul 28 16:10:57.933200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.932952 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbbm\" (UniqueName: \"kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm\") pod \"23c5edca-62f6-4190-b221-a8293cb267c1\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " Jul 28 16:10:57.933200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.932998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location\") pod \"23c5edca-62f6-4190-b221-a8293cb267c1\" (UID: \"23c5edca-62f6-4190-b221-a8293cb267c1\") " Jul 28 16:10:57.933323 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.933299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23c5edca-62f6-4190-b221-a8293cb267c1" (UID: "23c5edca-62f6-4190-b221-a8293cb267c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:10:57.933415 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.933350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23c5edca-62f6-4190-b221-a8293cb267c1" (UID: "23c5edca-62f6-4190-b221-a8293cb267c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:10:57.933415 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.933359 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23c5edca-62f6-4190-b221-a8293cb267c1" (UID: "23c5edca-62f6-4190-b221-a8293cb267c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:10:57.935049 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.935028 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23c5edca-62f6-4190-b221-a8293cb267c1" (UID: "23c5edca-62f6-4190-b221-a8293cb267c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:10:57.935150 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:57.935134 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm" (OuterVolumeSpecName: "kube-api-access-4bbbm") pod "23c5edca-62f6-4190-b221-a8293cb267c1" (UID: "23c5edca-62f6-4190-b221-a8293cb267c1"). InnerVolumeSpecName "kube-api-access-4bbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:10:58.034476 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.034434 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:10:58.034476 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.034468 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23c5edca-62f6-4190-b221-a8293cb267c1-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:10:58.034476 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.034482 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23c5edca-62f6-4190-b221-a8293cb267c1-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:10:58.034702 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.034495 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbbm\" (UniqueName: \"kubernetes.io/projected/23c5edca-62f6-4190-b221-a8293cb267c1-kube-api-access-4bbbm\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:10:58.034702 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.034506 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23c5edca-62f6-4190-b221-a8293cb267c1-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:10:58.360311 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.360267 2575 generic.go:356] "Generic (PLEG): container finished" podID="23c5edca-62f6-4190-b221-a8293cb267c1" containerID="161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60" exitCode=0 Jul 28 16:10:58.360742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.360345 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" Jul 28 16:10:58.360742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.360349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerDied","Data":"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60"} Jul 28 16:10:58.360742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.360429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9" event={"ID":"23c5edca-62f6-4190-b221-a8293cb267c1","Type":"ContainerDied","Data":"37fdcaf20f36069c4edfe1e31986fdf2779499ec36d4fbff8c6d6f5de4cd6b5b"} Jul 28 16:10:58.360742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.360458 2575 scope.go:117] "RemoveContainer" containerID="c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969" Jul 28 16:10:58.370085 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.370058 2575 scope.go:117] "RemoveContainer" containerID="161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60" Jul 28 16:10:58.383695 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.383674 2575 scope.go:117] "RemoveContainer" containerID="afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252" Jul 28 16:10:58.390176 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390150 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:10:58.390472 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="storage-initializer" Jul 28 16:10:58.390472 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390473 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="storage-initializer" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390483 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390489 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390499 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kube-rbac-proxy" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390505 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kube-rbac-proxy" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390548 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kserve-container" Jul 28 16:10:58.390559 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.390555 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" containerName="kube-rbac-proxy" Jul 28 16:10:58.392512 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.392490 2575 scope.go:117] "RemoveContainer" containerID="c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969" Jul 28 16:10:58.392851 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:10:58.392826 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969\": container with ID starting with c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969 not found: ID does not exist" containerID="c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969" Jul 28 16:10:58.392951 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.392862 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969"} err="failed to get container status \"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969\": rpc error: code = NotFound desc = could not find container \"c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969\": container with ID starting with c3fb099bc47bc2ce6f5fd4e3bee8629a8ee57ad87a13eb675a3759cb1b4b9969 not found: ID does not exist" Jul 28 16:10:58.392951 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.392886 2575 scope.go:117] "RemoveContainer" containerID="161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60" Jul 28 16:10:58.393183 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:10:58.393165 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60\": container with ID starting with 161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60 not found: ID does not exist" containerID="161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60" Jul 28 16:10:58.393230 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.393190 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60"} err="failed to get container status \"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60\": rpc error: code = NotFound desc = could not find container \"161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60\": container with ID starting with 161c5e77f80dc7225311eb10697eaf373ad6754752adf91ee94bd2a7bf36ea60 not found: ID does not exist" Jul 28 16:10:58.393230 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.393207 2575 scope.go:117] "RemoveContainer" containerID="afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252" Jul 28 16:10:58.393468 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:10:58.393446 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252\": container with ID starting with afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252 not found: ID does not exist" containerID="afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252" Jul 28 16:10:58.393546 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.393474 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252"} err="failed to get container status \"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252\": rpc error: code = NotFound desc = could not find container \"afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252\": container with ID starting with afa609bd43d824bd52224eeac44798f6a512deeff0e7f34e2f3639b396da8252 not found: ID does not exist" Jul 28 16:10:58.393696 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.393678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.393925 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.393837 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:10:58.395547 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.395525 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:10:58.395880 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.395862 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:10:58.395964 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.395867 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:10:58.396087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.396033 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9qmk9"] Jul 28 16:10:58.396087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.396072 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:10:58.396241 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.396115 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:10:58.396241 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.396151 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\"" Jul 28 16:10:58.396498 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.396483 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\"" Jul 28 16:10:58.404712 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.404693 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:10:58.539202 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.539150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.539202 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.539218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.539518 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.539317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.539518 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.539354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.539518 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.539417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640341 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640341 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640341 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640771 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640771 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.640771 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.640748 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.641077 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.641058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.641164 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.641151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.642891 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.642852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.648668 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.648639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.704261 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.704218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:10:58.834981 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:58.834954 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:10:58.837599 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:10:58.837572 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663bc45d_3105_4184_8336_494c83070f74.slice/crio-8bf943d40b707465bb885c254916651ab27264a1b6731d72ddb94933ae6f8e35 WatchSource:0}: Error finding container 8bf943d40b707465bb885c254916651ab27264a1b6731d72ddb94933ae6f8e35: Status 404 returned error can't find the container with id 8bf943d40b707465bb885c254916651ab27264a1b6731d72ddb94933ae6f8e35 Jul 28 16:10:59.365862 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:59.365818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerStarted","Data":"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663"} Jul 28 16:10:59.366323 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:10:59.365897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerStarted","Data":"8bf943d40b707465bb885c254916651ab27264a1b6731d72ddb94933ae6f8e35"} Jul 28 16:11:00.372193 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:00.372157 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c5edca-62f6-4190-b221-a8293cb267c1" path="/var/lib/kubelet/pods/23c5edca-62f6-4190-b221-a8293cb267c1/volumes" Jul 28 16:11:03.384369 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:03.384336 2575 generic.go:356] "Generic (PLEG): container finished" podID="663bc45d-3105-4184-8336-494c83070f74" containerID="65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663" exitCode=0 Jul 28 16:11:03.384750 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:03.384424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerDied","Data":"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663"} Jul 28 16:11:04.390018 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.389984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerStarted","Data":"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1"} Jul 28 16:11:04.390461 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.390046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerStarted","Data":"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107"} Jul 28 16:11:04.390461 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.390314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:11:04.390572 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.390480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:11:04.391677 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.391648 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:04.409487 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:04.409432 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podStartSLOduration=2.52589415 podStartE2EDuration="6.409416293s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.883522143s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:11:04.408967784 +0000 UTC m=+2282.609815276" watchObservedRunningTime="2026-07-28 16:11:04.409416293 +0000 UTC m=+2282.610263781" Jul 28 16:11:05.393760 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:05.393710 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:06.397564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:06.397524 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:11.403062 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:11.403033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:11:11.403743 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:11.403711 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:21.404421 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:21.404363 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:31.403819 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:31.403775 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:41.404205 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:41.404164 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:11:51.404081 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:11:51.404041 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:12:01.403792 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:01.403751 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:12:11.404238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:11.404195 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:12:21.404730 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:21.404700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:12:28.692385 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:28.692351 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:12:28.692834 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:28.692617 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" containerID="cri-o://8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107" gracePeriod=30 Jul 28 16:12:28.692834 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:28.692683 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kube-rbac-proxy" containerID="cri-o://9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1" gracePeriod=30 Jul 28 16:12:29.695234 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:29.695199 2575 generic.go:356] "Generic (PLEG): container finished" podID="663bc45d-3105-4184-8336-494c83070f74" containerID="9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1" exitCode=2 Jul 28 16:12:29.695633 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:29.695272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerDied","Data":"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1"} Jul 28 16:12:31.398549 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:31.398501 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused" Jul 28 16:12:31.403985 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:31.403945 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 28 16:12:33.430442 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.430415 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:12:33.564690 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.564657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls\") pod \"663bc45d-3105-4184-8336-494c83070f74\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " Jul 28 16:12:33.564863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.564705 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert\") pod \"663bc45d-3105-4184-8336-494c83070f74\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " Jul 28 16:12:33.564863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.564745 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location\") pod \"663bc45d-3105-4184-8336-494c83070f74\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " Jul 28 16:12:33.564863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.564784 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config\") pod \"663bc45d-3105-4184-8336-494c83070f74\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " Jul 28 16:12:33.564863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.564801 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt\") pod \"663bc45d-3105-4184-8336-494c83070f74\" (UID: \"663bc45d-3105-4184-8336-494c83070f74\") " Jul 28 16:12:33.565133 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.565105 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "663bc45d-3105-4184-8336-494c83070f74" (UID: "663bc45d-3105-4184-8336-494c83070f74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:12:33.565200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.565116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "663bc45d-3105-4184-8336-494c83070f74" (UID: "663bc45d-3105-4184-8336-494c83070f74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:12:33.565250 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.565203 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "663bc45d-3105-4184-8336-494c83070f74" (UID: "663bc45d-3105-4184-8336-494c83070f74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:12:33.566857 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.566831 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "663bc45d-3105-4184-8336-494c83070f74" (UID: "663bc45d-3105-4184-8336-494c83070f74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:12:33.566953 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.566926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt" (OuterVolumeSpecName: "kube-api-access-xmwvt") pod "663bc45d-3105-4184-8336-494c83070f74" (UID: "663bc45d-3105-4184-8336-494c83070f74"). InnerVolumeSpecName "kube-api-access-xmwvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:12:33.665690 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.665655 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:12:33.665690 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.665685 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/663bc45d-3105-4184-8336-494c83070f74-kube-api-access-xmwvt\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:12:33.665690 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.665696 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663bc45d-3105-4184-8336-494c83070f74-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:12:33.665922 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.665705 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663bc45d-3105-4184-8336-494c83070f74-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:12:33.665922 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.665715 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663bc45d-3105-4184-8336-494c83070f74-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:12:33.712976 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.712938 2575 generic.go:356] "Generic (PLEG): container finished" podID="663bc45d-3105-4184-8336-494c83070f74" containerID="8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107" exitCode=0 Jul 28 16:12:33.713131 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.712986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerDied","Data":"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107"} Jul 28 16:12:33.713131 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.712991 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" Jul 28 16:12:33.713131 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.713036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb" event={"ID":"663bc45d-3105-4184-8336-494c83070f74","Type":"ContainerDied","Data":"8bf943d40b707465bb885c254916651ab27264a1b6731d72ddb94933ae6f8e35"} Jul 28 16:12:33.713131 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.713062 2575 scope.go:117] "RemoveContainer" containerID="9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1" Jul 28 16:12:33.721749 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.721720 2575 scope.go:117] "RemoveContainer" containerID="8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107" Jul 28 16:12:33.730191 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.730171 2575 scope.go:117] "RemoveContainer" containerID="65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663" Jul 28 16:12:33.736328 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.736305 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:12:33.737507 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.737490 2575 scope.go:117] "RemoveContainer" containerID="9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1" Jul 28 16:12:33.737804 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:12:33.737783 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1\": container with ID starting with 9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1 not found: ID does not exist" containerID="9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1" Jul 28 16:12:33.737849 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.737809 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1"} err="failed to get container status \"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1\": rpc error: code = NotFound desc = could not find container \"9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1\": container with ID starting with 9b5423cf1834367abccb546bcee56f26bbefe9610488066028123e82154494c1 not found: ID does not exist" Jul 28 16:12:33.737849 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.737826 2575 scope.go:117] "RemoveContainer" containerID="8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107" Jul 28 16:12:33.738080 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:12:33.738061 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107\": container with ID starting with 8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107 not found: ID does not exist" containerID="8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107" Jul 28 16:12:33.738122 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.738088 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107"} err="failed to get container status \"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107\": rpc error: code = NotFound desc = could not find container \"8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107\": container with ID starting with 8033751d2791a96a29c376c168ee7a24ea70755ef3c9622b58ebad897b34b107 not found: ID does not exist" Jul 28 16:12:33.738122 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.738104 2575 scope.go:117] "RemoveContainer" containerID="65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663" Jul 28 16:12:33.738320 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:12:33.738303 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663\": container with ID starting with 65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663 not found: ID does not exist" containerID="65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663" Jul 28 16:12:33.738379 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.738326 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663"} err="failed to get container status \"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663\": rpc error: code = NotFound desc = could not find container \"65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663\": container with ID starting with 65f548d7a7f40f5f37baaa57f150bfb7e2dd778ea0ae6aede330dc33d8b80663 not found: ID does not exist" Jul 28 16:12:33.740974 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:33.740954 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zmtpb"] Jul 28 16:12:34.371621 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:12:34.371589 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663bc45d-3105-4184-8336-494c83070f74" path="/var/lib/kubelet/pods/663bc45d-3105-4184-8336-494c83070f74/volumes" Jul 28 16:15:39.242965 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.242928 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243186 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="storage-initializer" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243201 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="storage-initializer" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243219 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243228 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243243 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kube-rbac-proxy" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243250 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kube-rbac-proxy" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243296 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kserve-container" Jul 28 16:15:39.244613 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.243303 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="663bc45d-3105-4184-8336-494c83070f74" containerName="kube-rbac-proxy" Jul 28 16:15:39.245183 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.245164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.247345 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.247307 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\"" Jul 28 16:15:39.247509 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.247478 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:15:39.248036 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.248012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:15:39.248135 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.248035 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:15:39.248135 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.248013 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:15:39.248135 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.248013 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:15:39.248299 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.248207 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\"" Jul 28 16:15:39.256527 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.256504 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:15:39.335632 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.335595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.335810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.335636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.335810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.335662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.335810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.335722 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.335810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.335738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtqg\" (UniqueName: \"kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.436743 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.436695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.436743 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.436746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtqg\" (UniqueName: \"kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.436997 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.436793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.436997 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.436816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.436997 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.436836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.437192 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.437168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.437488 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.437467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.437545 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.437471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.439379 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.439353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.447010 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.446986 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtqg\" (UniqueName: \"kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.555195 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.555150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:39.684478 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:39.684447 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:15:39.687111 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:15:39.687084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d91dff_e988_42c6_a217_57b7ddbc578e.slice/crio-542db87648f5f9406dbb7f3de9902f082e6013265436850857c3d63fe8ebce27 WatchSource:0}: Error finding container 542db87648f5f9406dbb7f3de9902f082e6013265436850857c3d63fe8ebce27: Status 404 returned error can't find the container with id 542db87648f5f9406dbb7f3de9902f082e6013265436850857c3d63fe8ebce27 Jul 28 16:15:40.360297 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:40.360255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerStarted","Data":"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2"} Jul 28 16:15:40.360795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:40.360319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerStarted","Data":"542db87648f5f9406dbb7f3de9902f082e6013265436850857c3d63fe8ebce27"} Jul 28 16:15:44.374505 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:44.374471 2575 generic.go:356] "Generic (PLEG): container finished" podID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerID="c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2" exitCode=0 Jul 28 16:15:44.374861 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:44.374545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerDied","Data":"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2"} Jul 28 16:15:45.380841 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:45.380800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerStarted","Data":"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4"} Jul 28 16:15:45.381289 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:45.380884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerStarted","Data":"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991"} Jul 28 16:15:45.381289 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:45.381201 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:45.407048 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:45.406990 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" podStartSLOduration=2.436918535 podStartE2EDuration="6.406974369s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.970055834s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:15:45.404873756 +0000 UTC m=+2563.605721247" watchObservedRunningTime="2026-07-28 16:15:45.406974369 +0000 UTC m=+2563.607821862" Jul 28 16:15:46.384072 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:46.384040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:46.385555 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:46.385519 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 28 16:15:47.387128 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:47.387084 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 28 16:15:52.391452 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:52.391417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:52.392445 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:52.392423 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:15:59.282300 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:59.282261 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:15:59.282735 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:59.282517 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" containerID="cri-o://e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991" gracePeriod=30 Jul 28 16:15:59.282735 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:59.282639 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kube-rbac-proxy" containerID="cri-o://47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4" gracePeriod=30 Jul 28 16:15:59.431115 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:59.431078 2575 generic.go:356] "Generic (PLEG): container finished" podID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerID="47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4" exitCode=2 Jul 28 16:15:59.431286 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:15:59.431150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerDied","Data":"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4"} Jul 28 16:16:00.014976 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.014951 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:16:00.101619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101537 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gtqg\" (UniqueName: \"kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg\") pod \"56d91dff-e988-42c6-a217-57b7ddbc578e\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " Jul 28 16:16:00.101619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101593 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert\") pod \"56d91dff-e988-42c6-a217-57b7ddbc578e\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " Jul 28 16:16:00.101619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101613 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location\") pod \"56d91dff-e988-42c6-a217-57b7ddbc578e\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " Jul 28 16:16:00.101890 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101653 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config\") pod \"56d91dff-e988-42c6-a217-57b7ddbc578e\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " Jul 28 16:16:00.101890 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101685 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls\") pod \"56d91dff-e988-42c6-a217-57b7ddbc578e\" (UID: \"56d91dff-e988-42c6-a217-57b7ddbc578e\") " Jul 28 16:16:00.101998 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.101955 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56d91dff-e988-42c6-a217-57b7ddbc578e" (UID: "56d91dff-e988-42c6-a217-57b7ddbc578e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:16:00.102055 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.102024 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56d91dff-e988-42c6-a217-57b7ddbc578e" (UID: "56d91dff-e988-42c6-a217-57b7ddbc578e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:16:00.102095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.102059 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56d91dff-e988-42c6-a217-57b7ddbc578e" (UID: "56d91dff-e988-42c6-a217-57b7ddbc578e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:16:00.103775 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.103745 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56d91dff-e988-42c6-a217-57b7ddbc578e" (UID: "56d91dff-e988-42c6-a217-57b7ddbc578e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:16:00.103887 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.103826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg" (OuterVolumeSpecName: "kube-api-access-9gtqg") pod "56d91dff-e988-42c6-a217-57b7ddbc578e" (UID: "56d91dff-e988-42c6-a217-57b7ddbc578e"). InnerVolumeSpecName "kube-api-access-9gtqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:16:00.202574 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.202517 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:16:00.202574 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.202564 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d91dff-e988-42c6-a217-57b7ddbc578e-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:16:00.202574 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.202579 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gtqg\" (UniqueName: \"kubernetes.io/projected/56d91dff-e988-42c6-a217-57b7ddbc578e-kube-api-access-9gtqg\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:16:00.202924 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.202594 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d91dff-e988-42c6-a217-57b7ddbc578e-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:16:00.202924 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.202606 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d91dff-e988-42c6-a217-57b7ddbc578e-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:16:00.436781 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.436746 2575 generic.go:356] "Generic (PLEG): container finished" podID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerID="e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991" exitCode=0 Jul 28 16:16:00.437238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.436795 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerDied","Data":"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991"} Jul 28 16:16:00.437238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.436867 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" event={"ID":"56d91dff-e988-42c6-a217-57b7ddbc578e","Type":"ContainerDied","Data":"542db87648f5f9406dbb7f3de9902f082e6013265436850857c3d63fe8ebce27"} Jul 28 16:16:00.437238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.436808 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26" Jul 28 16:16:00.437238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.436908 2575 scope.go:117] "RemoveContainer" containerID="47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4" Jul 28 16:16:00.445086 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.445064 2575 scope.go:117] "RemoveContainer" containerID="e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991" Jul 28 16:16:00.454716 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.454691 2575 scope.go:117] "RemoveContainer" containerID="c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2" Jul 28 16:16:00.456413 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.456378 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:16:00.461708 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.461685 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mlv26"] Jul 28 16:16:00.462825 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.462809 2575 scope.go:117] "RemoveContainer" containerID="47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4" Jul 28 16:16:00.463192 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:16:00.463174 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4\": container with ID starting with 47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4 not found: ID does not exist" containerID="47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4" Jul 28 16:16:00.463288 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.463198 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4"} err="failed to get container status \"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4\": rpc error: code = NotFound desc = could not find container \"47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4\": container with ID starting with 47bf010bd26acfbf2b9286fa6f6149d5b6f6e8033b0e97475da9fb8fff8294b4 not found: ID does not exist" Jul 28 16:16:00.463288 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.463217 2575 scope.go:117] "RemoveContainer" containerID="e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991" Jul 28 16:16:00.463490 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:16:00.463473 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991\": container with ID starting with e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991 not found: ID does not exist" containerID="e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991" Jul 28 16:16:00.463526 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.463495 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991"} err="failed to get container status \"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991\": rpc error: code = NotFound desc = could not find container \"e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991\": container with ID starting with e41a53130e8b3c65645e32162799575674f12bbbac4a95fa4a8517a6cf4f4991 not found: ID does not exist" Jul 28 16:16:00.463526 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.463512 2575 scope.go:117] "RemoveContainer" containerID="c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2" Jul 28 16:16:00.463756 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:16:00.463739 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2\": container with ID starting with c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2 not found: ID does not exist" containerID="c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2" Jul 28 16:16:00.463805 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:00.463761 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2"} err="failed to get container status \"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2\": rpc error: code = NotFound desc = could not find container \"c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2\": container with ID starting with c4bc1aec62dc4316a959ab480de80ed250c3e03adfb05ca776df79101f66fba2 not found: ID does not exist" Jul 28 16:16:02.373025 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:16:02.372992 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" path="/var/lib/kubelet/pods/56d91dff-e988-42c6-a217-57b7ddbc578e/volumes" Jul 28 16:18:25.043138 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043100 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043351 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043362 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043371 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="storage-initializer" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043377 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="storage-initializer" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043385 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kube-rbac-proxy" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043403 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kube-rbac-proxy" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043445 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kserve-container" Jul 28 16:18:25.043664 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.043455 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d91dff-e988-42c6-a217-57b7ddbc578e" containerName="kube-rbac-proxy" Jul 28 16:18:25.046209 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.046189 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.048721 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.048514 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\"" Jul 28 16:18:25.048721 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.048578 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\"" Jul 28 16:18:25.048721 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.048607 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:18:25.048721 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.048635 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:18:25.054378 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.049182 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:18:25.054378 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.049453 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:18:25.054378 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.049770 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:18:25.057656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.057631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:18:25.195210 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.195172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.195384 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.195217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.195384 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.195308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.195384 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.195345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.195535 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.195403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6kl5\" (UniqueName: \"kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296009 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.295913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296009 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.295955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296009 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.295984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296277 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.296027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296277 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.296069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6kl5\" (UniqueName: \"kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.296376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.296721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.296850 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.296833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.298602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.298586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.305217 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.305190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6kl5\" (UniqueName: \"kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-zvbcs\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.362233 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.362195 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:25.495480 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.495360 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:18:25.498280 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:18:25.498256 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26509a87_fb40_4106_a215_93d94fd3813a.slice/crio-1e210e8d791b47f2caa93514e86fbb6975970e41d99c06a2fb796bb285e91e1d WatchSource:0}: Error finding container 1e210e8d791b47f2caa93514e86fbb6975970e41d99c06a2fb796bb285e91e1d: Status 404 returned error can't find the container with id 1e210e8d791b47f2caa93514e86fbb6975970e41d99c06a2fb796bb285e91e1d Jul 28 16:18:25.932317 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.932224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerStarted","Data":"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64"} Jul 28 16:18:25.932317 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:25.932296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerStarted","Data":"1e210e8d791b47f2caa93514e86fbb6975970e41d99c06a2fb796bb285e91e1d"} Jul 28 16:18:30.952292 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:30.952250 2575 generic.go:356] "Generic (PLEG): container finished" podID="26509a87-fb40-4106-a215-93d94fd3813a" containerID="2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64" exitCode=0 Jul 28 16:18:30.952692 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:30.952322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerDied","Data":"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64"} Jul 28 16:18:30.953537 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:30.953522 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 16:18:43.000748 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:43.000713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerStarted","Data":"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b"} Jul 28 16:18:43.001261 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:43.000784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerStarted","Data":"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee"} Jul 28 16:18:43.001261 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:43.001010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:43.022152 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:43.022097 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podStartSLOduration=2.178395259 podStartE2EDuration="18.022079602s" totalImagesPullingTime="11.1443951s" totalInitContainerRuntime="4.699289243s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:18:43.020300756 +0000 UTC m=+2741.221148244" watchObservedRunningTime="2026-07-28 16:18:43.022079602 +0000 UTC m=+2741.222927093" Jul 28 16:18:44.009203 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:44.007546 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:44.009203 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:44.007793 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:18:45.009180 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:45.009133 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:18:50.014171 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:50.014140 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:18:50.014711 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:18:50.014680 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:19:00.014687 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:00.014649 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:19:10.014773 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:10.014725 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:19:20.015461 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:20.015417 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 28 16:19:30.015937 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:30.015852 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:19:36.797715 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:36.797675 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:19:36.798182 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:36.797908 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" containerID="cri-o://4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee" gracePeriod=30 Jul 28 16:19:36.798182 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:36.797972 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kube-rbac-proxy" containerID="cri-o://3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b" gracePeriod=30 Jul 28 16:19:37.192579 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:37.192479 2575 generic.go:356] "Generic (PLEG): container finished" podID="26509a87-fb40-4106-a215-93d94fd3813a" containerID="3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b" exitCode=2 Jul 28 16:19:37.192579 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:37.192535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerDied","Data":"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b"} Jul 28 16:19:39.636307 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.636282 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:19:39.785795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.785751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config\") pod \"26509a87-fb40-4106-a215-93d94fd3813a\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " Jul 28 16:19:39.785955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.785819 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert\") pod \"26509a87-fb40-4106-a215-93d94fd3813a\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " Jul 28 16:19:39.785955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.785859 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls\") pod \"26509a87-fb40-4106-a215-93d94fd3813a\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " Jul 28 16:19:39.785955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.785905 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6kl5\" (UniqueName: \"kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5\") pod \"26509a87-fb40-4106-a215-93d94fd3813a\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " Jul 28 16:19:39.786054 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.785968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location\") pod \"26509a87-fb40-4106-a215-93d94fd3813a\" (UID: \"26509a87-fb40-4106-a215-93d94fd3813a\") " Jul 28 16:19:39.786237 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.786213 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26509a87-fb40-4106-a215-93d94fd3813a" (UID: "26509a87-fb40-4106-a215-93d94fd3813a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:19:39.786344 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.786270 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26509a87-fb40-4106-a215-93d94fd3813a" (UID: "26509a87-fb40-4106-a215-93d94fd3813a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:19:39.788133 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.788102 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26509a87-fb40-4106-a215-93d94fd3813a" (UID: "26509a87-fb40-4106-a215-93d94fd3813a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:19:39.788239 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.788176 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5" (OuterVolumeSpecName: "kube-api-access-d6kl5") pod "26509a87-fb40-4106-a215-93d94fd3813a" (UID: "26509a87-fb40-4106-a215-93d94fd3813a"). InnerVolumeSpecName "kube-api-access-d6kl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:19:39.795671 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.795641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26509a87-fb40-4106-a215-93d94fd3813a" (UID: "26509a87-fb40-4106-a215-93d94fd3813a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:19:39.886515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.886462 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:19:39.886515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.886508 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26509a87-fb40-4106-a215-93d94fd3813a-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:19:39.886515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.886517 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26509a87-fb40-4106-a215-93d94fd3813a-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:19:39.886515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.886526 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6kl5\" (UniqueName: \"kubernetes.io/projected/26509a87-fb40-4106-a215-93d94fd3813a-kube-api-access-d6kl5\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:19:39.886791 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:39.886537 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26509a87-fb40-4106-a215-93d94fd3813a-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:19:40.204228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.204134 2575 generic.go:356] "Generic (PLEG): container finished" podID="26509a87-fb40-4106-a215-93d94fd3813a" containerID="4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee" exitCode=0 Jul 28 16:19:40.204228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.204183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerDied","Data":"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee"} Jul 28 16:19:40.204228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.204191 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" Jul 28 16:19:40.204569 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.204235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs" event={"ID":"26509a87-fb40-4106-a215-93d94fd3813a","Type":"ContainerDied","Data":"1e210e8d791b47f2caa93514e86fbb6975970e41d99c06a2fb796bb285e91e1d"} Jul 28 16:19:40.204569 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.204270 2575 scope.go:117] "RemoveContainer" containerID="3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b" Jul 28 16:19:40.212451 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.212428 2575 scope.go:117] "RemoveContainer" containerID="4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee" Jul 28 16:19:40.221007 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.220967 2575 scope.go:117] "RemoveContainer" containerID="2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64" Jul 28 16:19:40.225989 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.225965 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:19:40.228687 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.228662 2575 scope.go:117] "RemoveContainer" containerID="3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b" Jul 28 16:19:40.229190 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:19:40.229110 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b\": container with ID starting with 3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b not found: ID does not exist" containerID="3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b" Jul 28 16:19:40.229190 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.229143 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b"} err="failed to get container status \"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b\": rpc error: code = NotFound desc = could not find container \"3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b\": container with ID starting with 3c2938b8af7bbf5299beb9f57a519854d9b192ccecd1dcf8afb599f54db5573b not found: ID does not exist" Jul 28 16:19:40.229190 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.229168 2575 scope.go:117] "RemoveContainer" containerID="4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee" Jul 28 16:19:40.229463 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:19:40.229443 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee\": container with ID starting with 4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee not found: ID does not exist" containerID="4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee" Jul 28 16:19:40.229535 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.229470 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee"} err="failed to get container status \"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee\": rpc error: code = NotFound desc = could not find container \"4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee\": container with ID starting with 4c57cf83d6ac9d84cf259229001b7418040985028dc99db4d777287e1cb853ee not found: ID does not exist" Jul 28 16:19:40.229535 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.229487 2575 scope.go:117] "RemoveContainer" containerID="2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64" Jul 28 16:19:40.229744 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:19:40.229725 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64\": container with ID starting with 2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64 not found: ID does not exist" containerID="2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64" Jul 28 16:19:40.229783 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.229752 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64"} err="failed to get container status \"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64\": rpc error: code = NotFound desc = could not find container \"2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64\": container with ID starting with 2eb450208db60d66fd0f360d3fc0d801003406c948189bc454975ac5ee7cfb64 not found: ID does not exist" Jul 28 16:19:40.230196 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.230178 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-zvbcs"] Jul 28 16:19:40.371624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:19:40.371593 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26509a87-fb40-4106-a215-93d94fd3813a" path="/var/lib/kubelet/pods/26509a87-fb40-4106-a215-93d94fd3813a/volumes" Jul 28 16:29:41.462654 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462620 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462866 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kube-rbac-proxy" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462879 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kube-rbac-proxy" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462895 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="storage-initializer" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462901 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="storage-initializer" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462908 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462913 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462952 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kserve-container" Jul 28 16:29:41.463068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.462959 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26509a87-fb40-4106-a215-93d94fd3813a" containerName="kube-rbac-proxy" Jul 28 16:29:41.464928 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.464912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.467108 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467080 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-08e8a1\"" Jul 28 16:29:41.467307 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:29:41.467307 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467208 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-08e8a1-predictor-serving-cert\"" Jul 28 16:29:41.467307 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-08e8a1-dockercfg-kn74w\"" Jul 28 16:29:41.467534 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467489 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:29:41.467722 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467704 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-08e8a1-kube-rbac-proxy-sar-config\"" Jul 28 16:29:41.467722 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.467716 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:29:41.474992 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.474967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:41.599851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.599810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.599851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.599855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.600074 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.599879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.600074 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.599902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29j9k\" (UniqueName: \"kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.600074 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.599958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.700731 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.700689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.700731 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.700738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.700995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.700885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.700995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.700924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29j9k\" (UniqueName: \"kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.700995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.700963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.701288 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.701269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.701497 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.701477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.701497 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.701492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.703339 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.703313 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.709425 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.709382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29j9k\" (UniqueName: \"kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k\") pod \"isvc-init-fail-08e8a1-predictor-947c689bb-jh76p\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.775494 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.775450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:41.908292 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:41.908260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:41.911300 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:29:41.911272 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6b4455_ef04_4423_b85f_15aa161e76f2.slice/crio-e4afde43a8ca952a0f4c2266f1484a393e36e63e476e1602570824c658e6dfd3 WatchSource:0}: Error finding container e4afde43a8ca952a0f4c2266f1484a393e36e63e476e1602570824c658e6dfd3: Status 404 returned error can't find the container with id e4afde43a8ca952a0f4c2266f1484a393e36e63e476e1602570824c658e6dfd3 Jul 28 16:29:42.252008 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:42.251922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerStarted","Data":"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a"} Jul 28 16:29:42.252008 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:42.251983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerStarted","Data":"e4afde43a8ca952a0f4c2266f1484a393e36e63e476e1602570824c658e6dfd3"} Jul 28 16:29:45.263533 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:45.263506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/0.log" Jul 28 16:29:45.263911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:45.263544 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerID="56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a" exitCode=1 Jul 28 16:29:45.263911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:45.263574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerDied","Data":"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a"} Jul 28 16:29:46.268872 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:46.268846 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/0.log" Jul 28 16:29:46.269263 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:46.268899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerStarted","Data":"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186"} Jul 28 16:29:46.504047 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:46.504013 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:47.272941 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:47.272903 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" containerID="cri-o://3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186" gracePeriod=30 Jul 28 16:29:48.213878 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.213854 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/1.log" Jul 28 16:29:48.214207 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.214193 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/0.log" Jul 28 16:29:48.216475 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.216461 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:48.277379 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.277349 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/1.log" Jul 28 16:29:48.277833 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.277812 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-08e8a1-predictor-947c689bb-jh76p_2e6b4455-ef04-4423-b85f-15aa161e76f2/storage-initializer/0.log" Jul 28 16:29:48.277910 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.277859 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerID="3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186" exitCode=1 Jul 28 16:29:48.277959 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.277916 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" Jul 28 16:29:48.277959 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.277943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerDied","Data":"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186"} Jul 28 16:29:48.278055 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.278041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p" event={"ID":"2e6b4455-ef04-4423-b85f-15aa161e76f2","Type":"ContainerDied","Data":"e4afde43a8ca952a0f4c2266f1484a393e36e63e476e1602570824c658e6dfd3"} Jul 28 16:29:48.278092 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.278074 2575 scope.go:117] "RemoveContainer" containerID="3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186" Jul 28 16:29:48.285877 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.285860 2575 scope.go:117] "RemoveContainer" containerID="56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a" Jul 28 16:29:48.293187 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.293169 2575 scope.go:117] "RemoveContainer" containerID="3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186" Jul 28 16:29:48.293480 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:29:48.293458 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186\": container with ID starting with 3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186 not found: ID does not exist" containerID="3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186" Jul 28 16:29:48.293579 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.293486 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186"} err="failed to get container status \"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186\": rpc error: code = NotFound desc = could not find container \"3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186\": container with ID starting with 3f7adbc56c7f55998f754d80e052b58fe077dab8eaf2a96b0e593c08eaa6b186 not found: ID does not exist" Jul 28 16:29:48.293579 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.293504 2575 scope.go:117] "RemoveContainer" containerID="56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a" Jul 28 16:29:48.293788 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:29:48.293770 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a\": container with ID starting with 56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a not found: ID does not exist" containerID="56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a" Jul 28 16:29:48.293828 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.293794 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a"} err="failed to get container status \"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a\": rpc error: code = NotFound desc = could not find container \"56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a\": container with ID starting with 56eec3bdf73edf980c795f9caac7a5566a4dfa377931f9c2a5456147de61869a not found: ID does not exist" Jul 28 16:29:48.353669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353585 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29j9k\" (UniqueName: \"kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k\") pod \"2e6b4455-ef04-4423-b85f-15aa161e76f2\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " Jul 28 16:29:48.353669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert\") pod \"2e6b4455-ef04-4423-b85f-15aa161e76f2\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " Jul 28 16:29:48.353669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353641 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location\") pod \"2e6b4455-ef04-4423-b85f-15aa161e76f2\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " Jul 28 16:29:48.353995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353687 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config\") pod \"2e6b4455-ef04-4423-b85f-15aa161e76f2\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " Jul 28 16:29:48.353995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls\") pod \"2e6b4455-ef04-4423-b85f-15aa161e76f2\" (UID: \"2e6b4455-ef04-4423-b85f-15aa161e76f2\") " Jul 28 16:29:48.353995 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.353938 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e6b4455-ef04-4423-b85f-15aa161e76f2" (UID: "2e6b4455-ef04-4423-b85f-15aa161e76f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:29:48.354155 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.354055 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e6b4455-ef04-4423-b85f-15aa161e76f2" (UID: "2e6b4455-ef04-4423-b85f-15aa161e76f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:29:48.354155 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.354107 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e6b4455-ef04-4423-b85f-15aa161e76f2" (UID: "2e6b4455-ef04-4423-b85f-15aa161e76f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:29:48.355704 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.355680 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k" (OuterVolumeSpecName: "kube-api-access-29j9k") pod "2e6b4455-ef04-4423-b85f-15aa161e76f2" (UID: "2e6b4455-ef04-4423-b85f-15aa161e76f2"). InnerVolumeSpecName "kube-api-access-29j9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:29:48.355824 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.355801 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e6b4455-ef04-4423-b85f-15aa161e76f2" (UID: "2e6b4455-ef04-4423-b85f-15aa161e76f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:29:48.454779 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.454738 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29j9k\" (UniqueName: \"kubernetes.io/projected/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-api-access-29j9k\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:29:48.454779 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.454767 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:29:48.454779 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.454776 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6b4455-ef04-4423-b85f-15aa161e76f2-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:29:48.454779 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.454785 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6b4455-ef04-4423-b85f-15aa161e76f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:29:48.455035 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.454795 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6b4455-ef04-4423-b85f-15aa161e76f2-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:29:48.609565 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.609487 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:48.613781 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:48.613756 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-08e8a1-predictor-947c689bb-jh76p"] Jul 28 16:29:50.372298 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:50.372264 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" path="/var/lib/kubelet/pods/2e6b4455-ef04-4423-b85f-15aa161e76f2/volumes" Jul 28 16:29:51.727200 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727165 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:29:51.727619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727468 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.727619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.727619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727542 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.727619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727597 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.727619 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727604 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.727805 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.727636 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6b4455-ef04-4423-b85f-15aa161e76f2" containerName="storage-initializer" Jul 28 16:29:51.732056 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.732037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.734791 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.734770 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:29:51.734920 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.734834 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:29:51.735028 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.735012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:29:51.735093 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.735054 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:29:51.735641 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.735626 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\"" Jul 28 16:29:51.735862 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.735836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:29:51.735977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.735866 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\"" Jul 28 16:29:51.741354 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.741332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:29:51.880611 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.880566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rth97\" (UniqueName: \"kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.880611 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.880612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.880892 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.880715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.880892 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.880769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.880892 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.880814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981297 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981297 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981602 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:29:51.981363 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found Jul 28 16:29:51.981602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rth97\" (UniqueName: \"kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981602 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:29:51.981456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls podName:bf28de63-8724-4009-afee-74929588bcff nodeName:}" failed. No retries permitted until 2026-07-28 16:29:52.481435796 +0000 UTC m=+3410.682283269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-kl827" (UID: "bf28de63-8724-4009-afee-74929588bcff") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found Jul 28 16:29:51.981602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981887 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.981978 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.981963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.982027 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.982010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:51.990232 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:51.990209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rth97\" (UniqueName: \"kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:52.484983 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:52.484940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:52.487468 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:52.487433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-kl827\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:52.643180 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:52.643138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:29:52.774311 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:52.774278 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:29:52.776572 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:29:52.776541 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf28de63_8724_4009_afee_74929588bcff.slice/crio-f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee WatchSource:0}: Error finding container f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee: Status 404 returned error can't find the container with id f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee Jul 28 16:29:53.299429 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:53.299378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerStarted","Data":"5cba0bc34257f22c65932cebecc94bf97bda17d5e40b0cc2a787963c57cc940d"} Jul 28 16:29:53.299604 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:53.299484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerStarted","Data":"f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee"} Jul 28 16:29:57.316806 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:57.316769 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf28de63-8724-4009-afee-74929588bcff" containerID="5cba0bc34257f22c65932cebecc94bf97bda17d5e40b0cc2a787963c57cc940d" exitCode=0 Jul 28 16:29:57.316806 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:57.316811 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerDied","Data":"5cba0bc34257f22c65932cebecc94bf97bda17d5e40b0cc2a787963c57cc940d"} Jul 28 16:29:57.317979 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:29:57.317963 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 16:30:18.413447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:18.413414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerStarted","Data":"40abb13a5099c27bad5d9c900b16ad9aa3874fb69828269b15a80f6bdbc4baf7"} Jul 28 16:30:18.413858 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:18.413476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerStarted","Data":"5586039f13f38836ac354b05c03af55ae532cd6124996a5d3fdf106ad26c4248"} Jul 28 16:30:18.413858 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:18.413662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:30:18.434278 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:18.434227 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podStartSLOduration=3.195083007 podStartE2EDuration="27.43421225s" totalImagesPullingTime="20.512670401s" totalInitContainerRuntime="3.726458842s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:18.433262483 +0000 UTC m=+3436.634109981" watchObservedRunningTime="2026-07-28 16:30:18.43421225 +0000 UTC m=+3436.635059741" Jul 28 16:30:19.417142 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:19.417108 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:30:19.418273 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:19.418245 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:30:20.420658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:20.420613 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:30:25.425194 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:25.425161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:30:25.425771 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:25.425740 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:30:35.426657 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:35.426620 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:30:45.426648 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:45.426610 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:30:55.425991 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:30:55.425951 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:31:05.426483 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:05.426440 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:31:15.426445 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:15.426403 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:31:25.426347 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:25.426260 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:31:35.426603 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:35.426568 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:31:41.890087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:41.890050 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:31:41.890624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:41.890307 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" containerID="cri-o://5586039f13f38836ac354b05c03af55ae532cd6124996a5d3fdf106ad26c4248" gracePeriod=30 Jul 28 16:31:41.890624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:41.890337 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kube-rbac-proxy" containerID="cri-o://40abb13a5099c27bad5d9c900b16ad9aa3874fb69828269b15a80f6bdbc4baf7" gracePeriod=30 Jul 28 16:31:42.709772 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:42.709740 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf28de63-8724-4009-afee-74929588bcff" containerID="40abb13a5099c27bad5d9c900b16ad9aa3874fb69828269b15a80f6bdbc4baf7" exitCode=2 Jul 28 16:31:42.709956 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:42.709792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerDied","Data":"40abb13a5099c27bad5d9c900b16ad9aa3874fb69828269b15a80f6bdbc4baf7"} Jul 28 16:31:45.421702 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:45.421660 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused" Jul 28 16:31:45.426033 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:45.426005 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Jul 28 16:31:46.725965 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.725934 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf28de63-8724-4009-afee-74929588bcff" containerID="5586039f13f38836ac354b05c03af55ae532cd6124996a5d3fdf106ad26c4248" exitCode=0 Jul 28 16:31:46.726299 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.726011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerDied","Data":"5586039f13f38836ac354b05c03af55ae532cd6124996a5d3fdf106ad26c4248"} Jul 28 16:31:46.726299 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.726067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" event={"ID":"bf28de63-8724-4009-afee-74929588bcff","Type":"ContainerDied","Data":"f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee"} Jul 28 16:31:46.726299 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.726088 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5a23ab9237f872d1986b446da32287370768237df030e7f9d07dbd6bcfa5eee" Jul 28 16:31:46.736719 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.736696 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:31:46.775983 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.775956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rth97\" (UniqueName: \"kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97\") pod \"bf28de63-8724-4009-afee-74929588bcff\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " Jul 28 16:31:46.776148 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.775994 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") pod \"bf28de63-8724-4009-afee-74929588bcff\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " Jul 28 16:31:46.776148 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config\") pod \"bf28de63-8724-4009-afee-74929588bcff\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " Jul 28 16:31:46.776148 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location\") pod \"bf28de63-8724-4009-afee-74929588bcff\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " Jul 28 16:31:46.776148 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776114 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert\") pod \"bf28de63-8724-4009-afee-74929588bcff\" (UID: \"bf28de63-8724-4009-afee-74929588bcff\") " Jul 28 16:31:46.776437 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776415 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf28de63-8724-4009-afee-74929588bcff" (UID: "bf28de63-8724-4009-afee-74929588bcff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:31:46.776504 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776451 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf28de63-8724-4009-afee-74929588bcff" (UID: "bf28de63-8724-4009-afee-74929588bcff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:31:46.776580 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.776553 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf28de63-8724-4009-afee-74929588bcff" (UID: "bf28de63-8724-4009-afee-74929588bcff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:31:46.778143 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.778123 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97" (OuterVolumeSpecName: "kube-api-access-rth97") pod "bf28de63-8724-4009-afee-74929588bcff" (UID: "bf28de63-8724-4009-afee-74929588bcff"). InnerVolumeSpecName "kube-api-access-rth97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:31:46.778220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.778148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf28de63-8724-4009-afee-74929588bcff" (UID: "bf28de63-8724-4009-afee-74929588bcff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:31:46.876713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.876629 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf28de63-8724-4009-afee-74929588bcff-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:31:46.876713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.876681 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:31:46.876713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.876691 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rth97\" (UniqueName: \"kubernetes.io/projected/bf28de63-8724-4009-afee-74929588bcff-kube-api-access-rth97\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:31:46.876713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.876699 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf28de63-8724-4009-afee-74929588bcff-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:31:46.876713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:46.876707 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf28de63-8724-4009-afee-74929588bcff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:31:47.069344 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069305 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:31:47.069618 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069604 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="storage-initializer" Jul 28 16:31:47.069676 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069620 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="storage-initializer" Jul 28 16:31:47.069676 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069630 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" Jul 28 16:31:47.069676 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069636 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" Jul 28 16:31:47.069676 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069643 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kube-rbac-proxy" Jul 28 16:31:47.069676 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069649 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kube-rbac-proxy" Jul 28 16:31:47.069833 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069689 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kube-rbac-proxy" Jul 28 16:31:47.069833 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.069698 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf28de63-8724-4009-afee-74929588bcff" containerName="kserve-container" Jul 28 16:31:47.071899 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.071882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.073790 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.073763 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\"" Jul 28 16:31:47.073929 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.073768 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\"" Jul 28 16:31:47.084803 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.084773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:31:47.178658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.178557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.178658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.178606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45pq\" (UniqueName: \"kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.178902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.178666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.178902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.178750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.178902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.178814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279162 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279162 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45pq\" (UniqueName: \"kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279429 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279498 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279551 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279834 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.279947 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.279926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.280121 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.280099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.281675 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.281659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.288074 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.288053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45pq\" (UniqueName: \"kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq\") pod \"isvc-predictive-xgboost-predictor-695f978978-vdfzn\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.383220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.383191 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:47.529681 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.529646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:31:47.531854 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:31:47.531825 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cd361d_d77c_478b_833b_6e19d2edb21f.slice/crio-f9bca2bab2353bbc8eec4c06fe5d25c6419083311c538f037713772165f6bf8e WatchSource:0}: Error finding container f9bca2bab2353bbc8eec4c06fe5d25c6419083311c538f037713772165f6bf8e: Status 404 returned error can't find the container with id f9bca2bab2353bbc8eec4c06fe5d25c6419083311c538f037713772165f6bf8e Jul 28 16:31:47.731732 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.731645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerStarted","Data":"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433"} Jul 28 16:31:47.731732 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.731697 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827" Jul 28 16:31:47.731732 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.731712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerStarted","Data":"f9bca2bab2353bbc8eec4c06fe5d25c6419083311c538f037713772165f6bf8e"} Jul 28 16:31:47.763324 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.763294 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:31:47.766186 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:47.766157 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-kl827"] Jul 28 16:31:48.372076 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:48.372040 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf28de63-8724-4009-afee-74929588bcff" path="/var/lib/kubelet/pods/bf28de63-8724-4009-afee-74929588bcff/volumes" Jul 28 16:31:51.749556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:51.749471 2575 generic.go:356] "Generic (PLEG): container finished" podID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerID="c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433" exitCode=0 Jul 28 16:31:51.749556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:51.749549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerDied","Data":"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433"} Jul 28 16:31:52.755611 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.755575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerStarted","Data":"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e"} Jul 28 16:31:52.756037 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.755634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerStarted","Data":"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0"} Jul 28 16:31:52.756037 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.755894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:52.756155 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.756035 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:52.757350 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.757322 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:31:52.778279 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:52.778234 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podStartSLOduration=2.0644031959999998 podStartE2EDuration="5.778220149s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713816953s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:52.777333985 +0000 UTC m=+3530.978181477" watchObservedRunningTime="2026-07-28 16:31:52.778220149 +0000 UTC m=+3530.979067640" Jul 28 16:31:53.760211 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:53.760175 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:31:58.765029 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:58.764993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:31:58.765597 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:31:58.765569 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:08.766203 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:08.766159 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:18.766315 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:18.766272 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:28.766263 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:28.766217 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:38.766229 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:38.766189 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:48.766451 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:48.766409 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:32:58.765682 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:32:58.765596 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:33:08.766537 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:08.766503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:33:17.224020 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:17.223983 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:33:17.224502 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:17.224220 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" containerID="cri-o://58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0" gracePeriod=30 Jul 28 16:33:17.224502 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:17.224267 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kube-rbac-proxy" containerID="cri-o://361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e" gracePeriod=30 Jul 28 16:33:18.064924 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:18.064886 2575 generic.go:356] "Generic (PLEG): container finished" podID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerID="361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e" exitCode=2 Jul 28 16:33:18.065095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:18.064937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerDied","Data":"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e"} Jul 28 16:33:18.760735 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:18.760687 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused" Jul 28 16:33:18.766077 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:18.766046 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Jul 28 16:33:22.066172 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.066145 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:33:22.080906 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.080871 2575 generic.go:356] "Generic (PLEG): container finished" podID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerID="58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0" exitCode=0 Jul 28 16:33:22.081088 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.080923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerDied","Data":"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0"} Jul 28 16:33:22.081088 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.080945 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" Jul 28 16:33:22.081088 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.080973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn" event={"ID":"06cd361d-d77c-478b-833b-6e19d2edb21f","Type":"ContainerDied","Data":"f9bca2bab2353bbc8eec4c06fe5d25c6419083311c538f037713772165f6bf8e"} Jul 28 16:33:22.081088 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.081006 2575 scope.go:117] "RemoveContainer" containerID="361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e" Jul 28 16:33:22.089624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.089600 2575 scope.go:117] "RemoveContainer" containerID="58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0" Jul 28 16:33:22.097213 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.097190 2575 scope.go:117] "RemoveContainer" containerID="c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433" Jul 28 16:33:22.104557 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.104535 2575 scope.go:117] "RemoveContainer" containerID="361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e" Jul 28 16:33:22.104799 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:33:22.104776 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e\": container with ID starting with 361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e not found: ID does not exist" containerID="361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e" Jul 28 16:33:22.104861 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.104810 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e"} err="failed to get container status \"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e\": rpc error: code = NotFound desc = could not find container \"361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e\": container with ID starting with 361728d0ebb1a66c5776781ffc4550554082e3c97e82dfe802a6b18d7026350e not found: ID does not exist" Jul 28 16:33:22.104861 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.104827 2575 scope.go:117] "RemoveContainer" containerID="58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0" Jul 28 16:33:22.105094 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:33:22.105068 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0\": container with ID starting with 58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0 not found: ID does not exist" containerID="58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0" Jul 28 16:33:22.105147 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.105104 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0"} err="failed to get container status \"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0\": rpc error: code = NotFound desc = could not find container \"58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0\": container with ID starting with 58a8ce6552b12010f41ff74689f63b9b9cd2a6e27a56d2d5b9eae97d943971e0 not found: ID does not exist" Jul 28 16:33:22.105147 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.105127 2575 scope.go:117] "RemoveContainer" containerID="c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433" Jul 28 16:33:22.105411 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:33:22.105375 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433\": container with ID starting with c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433 not found: ID does not exist" containerID="c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433" Jul 28 16:33:22.105701 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.105682 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433"} err="failed to get container status \"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433\": rpc error: code = NotFound desc = could not find container \"c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433\": container with ID starting with c1faeda233210c6025b019b548197e325a62ea449b90230e1542e4911e922433 not found: ID does not exist" Jul 28 16:33:22.135820 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.135790 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert\") pod \"06cd361d-d77c-478b-833b-6e19d2edb21f\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " Jul 28 16:33:22.135977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.135852 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config\") pod \"06cd361d-d77c-478b-833b-6e19d2edb21f\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " Jul 28 16:33:22.135977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.135875 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls\") pod \"06cd361d-d77c-478b-833b-6e19d2edb21f\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " Jul 28 16:33:22.135977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.135903 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location\") pod \"06cd361d-d77c-478b-833b-6e19d2edb21f\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " Jul 28 16:33:22.135977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.135927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v45pq\" (UniqueName: \"kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq\") pod \"06cd361d-d77c-478b-833b-6e19d2edb21f\" (UID: \"06cd361d-d77c-478b-833b-6e19d2edb21f\") " Jul 28 16:33:22.136248 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.136226 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06cd361d-d77c-478b-833b-6e19d2edb21f" (UID: "06cd361d-d77c-478b-833b-6e19d2edb21f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:33:22.136306 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.136266 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06cd361d-d77c-478b-833b-6e19d2edb21f" (UID: "06cd361d-d77c-478b-833b-6e19d2edb21f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:33:22.136306 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.136287 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06cd361d-d77c-478b-833b-6e19d2edb21f" (UID: "06cd361d-d77c-478b-833b-6e19d2edb21f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:33:22.138094 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.138067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06cd361d-d77c-478b-833b-6e19d2edb21f" (UID: "06cd361d-d77c-478b-833b-6e19d2edb21f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:33:22.138196 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.138104 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq" (OuterVolumeSpecName: "kube-api-access-v45pq") pod "06cd361d-d77c-478b-833b-6e19d2edb21f" (UID: "06cd361d-d77c-478b-833b-6e19d2edb21f"). InnerVolumeSpecName "kube-api-access-v45pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:33:22.237095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.236998 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:33:22.237095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.237029 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06cd361d-d77c-478b-833b-6e19d2edb21f-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:33:22.237095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.237040 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06cd361d-d77c-478b-833b-6e19d2edb21f-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:33:22.237095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.237049 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45pq\" (UniqueName: \"kubernetes.io/projected/06cd361d-d77c-478b-833b-6e19d2edb21f-kube-api-access-v45pq\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:33:22.237095 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.237058 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06cd361d-d77c-478b-833b-6e19d2edb21f-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:33:22.400040 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.400007 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:33:22.404492 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:22.404467 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-vdfzn"] Jul 28 16:33:24.378126 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:33:24.378088 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" path="/var/lib/kubelet/pods/06cd361d-d77c-478b-833b-6e19d2edb21f/volumes" Jul 28 16:36:02.573492 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:36:02.573462 2575 scope.go:117] "RemoveContainer" containerID="5cba0bc34257f22c65932cebecc94bf97bda17d5e40b0cc2a787963c57cc940d" Jul 28 16:37:02.587339 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:37:02.587309 2575 scope.go:117] "RemoveContainer" containerID="5586039f13f38836ac354b05c03af55ae532cd6124996a5d3fdf106ad26c4248" Jul 28 16:37:02.594871 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:37:02.594851 2575 scope.go:117] "RemoveContainer" containerID="40abb13a5099c27bad5d9c900b16ad9aa3874fb69828269b15a80f6bdbc4baf7" Jul 28 16:38:29.177123 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177085 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177326 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="storage-initializer" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177338 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="storage-initializer" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177356 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177363 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177369 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kube-rbac-proxy" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177376 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kube-rbac-proxy" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177437 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kube-rbac-proxy" Jul 28 16:38:29.177656 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.177446 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06cd361d-d77c-478b-833b-6e19d2edb21f" containerName="kserve-container" Jul 28 16:38:29.180256 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.180240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.183940 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.183912 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:38:29.184636 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.184622 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:38:29.184745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.184722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:38:29.185312 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.185292 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:38:29.185447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.185292 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\"" Jul 28 16:38:29.185447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.185295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\"" Jul 28 16:38:29.185447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.185362 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:38:29.193317 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.193295 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:38:29.217518 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.217484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.217684 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.217526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lss94\" (UniqueName: \"kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.217684 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.217624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.217684 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.217650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.217684 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.217683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319052 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319052 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lss94\" (UniqueName: \"kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319311 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319311 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319311 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319709 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319915 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.319915 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.319904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.321586 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.321560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.328234 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.328214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lss94\" (UniqueName: \"kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.491021 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.490910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:29.625048 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:29.624939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:38:29.627696 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:38:29.627658 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea95c3fa_7b40_411c_b59e_145971751437.slice/crio-66230aebd823be3ff7267e82c508f57707414a6a14aee0c49439adb3fad60548 WatchSource:0}: Error finding container 66230aebd823be3ff7267e82c508f57707414a6a14aee0c49439adb3fad60548: Status 404 returned error can't find the container with id 66230aebd823be3ff7267e82c508f57707414a6a14aee0c49439adb3fad60548 Jul 28 16:38:30.142710 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:30.142675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerStarted","Data":"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb"} Jul 28 16:38:30.142894 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:30.142742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerStarted","Data":"66230aebd823be3ff7267e82c508f57707414a6a14aee0c49439adb3fad60548"} Jul 28 16:38:34.158209 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:34.158176 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea95c3fa-7b40-411c-b59e-145971751437" containerID="13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb" exitCode=0 Jul 28 16:38:34.158607 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:34.158259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerDied","Data":"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb"} Jul 28 16:38:35.164228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:35.164183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerStarted","Data":"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2"} Jul 28 16:38:35.164949 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:35.164259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerStarted","Data":"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba"} Jul 28 16:38:35.164949 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:35.164562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:35.185363 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:35.185311 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podStartSLOduration=2.143017441 podStartE2EDuration="6.18529062s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.042273179s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:38:35.184639711 +0000 UTC m=+3933.385487196" watchObservedRunningTime="2026-07-28 16:38:35.18529062 +0000 UTC m=+3933.386138113" Jul 28 16:38:36.167725 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:36.167691 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:38:42.176457 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:38:42.176427 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:39:12.177540 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:12.177493 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" Jul 28 16:39:22.177179 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:22.177131 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" Jul 28 16:39:32.177028 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:32.176987 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" Jul 28 16:39:42.177405 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:42.177360 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" Jul 28 16:39:52.197550 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:52.197520 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:39:59.277337 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:59.277298 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:39:59.277873 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:59.277563 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" containerID="cri-o://9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba" gracePeriod=30 Jul 28 16:39:59.277873 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:59.277612 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kube-rbac-proxy" containerID="cri-o://74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2" gracePeriod=30 Jul 28 16:39:59.463015 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:59.462980 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea95c3fa-7b40-411c-b59e-145971751437" containerID="74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2" exitCode=2 Jul 28 16:39:59.463183 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:39:59.463023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerDied","Data":"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2"} Jul 28 16:40:02.171984 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:02.171940 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused" Jul 28 16:40:02.177466 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:02.177430 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" Jul 28 16:40:04.431100 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.431076 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:40:04.482733 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.482696 2575 generic.go:356] "Generic (PLEG): container finished" podID="ea95c3fa-7b40-411c-b59e-145971751437" containerID="9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba" exitCode=0 Jul 28 16:40:04.482902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.482764 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" Jul 28 16:40:04.482902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.482780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerDied","Data":"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba"} Jul 28 16:40:04.482902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.482842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd" event={"ID":"ea95c3fa-7b40-411c-b59e-145971751437","Type":"ContainerDied","Data":"66230aebd823be3ff7267e82c508f57707414a6a14aee0c49439adb3fad60548"} Jul 28 16:40:04.482902 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.482870 2575 scope.go:117] "RemoveContainer" containerID="74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2" Jul 28 16:40:04.490614 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.490595 2575 scope.go:117] "RemoveContainer" containerID="9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba" Jul 28 16:40:04.498402 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.498373 2575 scope.go:117] "RemoveContainer" containerID="13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb" Jul 28 16:40:04.507149 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507127 2575 scope.go:117] "RemoveContainer" containerID="74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2" Jul 28 16:40:04.507457 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:40:04.507434 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2\": container with ID starting with 74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2 not found: ID does not exist" containerID="74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2" Jul 28 16:40:04.507523 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507472 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2"} err="failed to get container status \"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2\": rpc error: code = NotFound desc = could not find container \"74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2\": container with ID starting with 74713abf24e4b1351ad6a78368c7d32c3bb96b1dc33ea9e6ca749a1508e134e2 not found: ID does not exist" Jul 28 16:40:04.507523 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507501 2575 scope.go:117] "RemoveContainer" containerID="9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba" Jul 28 16:40:04.507752 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:40:04.507733 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba\": container with ID starting with 9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba not found: ID does not exist" containerID="9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba" Jul 28 16:40:04.507810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507760 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba"} err="failed to get container status \"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba\": rpc error: code = NotFound desc = could not find container \"9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba\": container with ID starting with 9cefd0aa37d43df6ff699cddc8ce5c27e04fdd8da6160b5366ce39713c9e5cba not found: ID does not exist" Jul 28 16:40:04.507810 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507777 2575 scope.go:117] "RemoveContainer" containerID="13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb" Jul 28 16:40:04.507988 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:40:04.507974 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb\": container with ID starting with 13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb not found: ID does not exist" containerID="13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb" Jul 28 16:40:04.508045 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.507992 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb"} err="failed to get container status \"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb\": rpc error: code = NotFound desc = could not find container \"13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb\": container with ID starting with 13ad081cb887d3bfe431d4a7ecd5b5ecd031f5a724141c61a2a000b1677be9eb not found: ID does not exist" Jul 28 16:40:04.568380 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568284 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lss94\" (UniqueName: \"kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94\") pod \"ea95c3fa-7b40-411c-b59e-145971751437\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " Jul 28 16:40:04.568380 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert\") pod \"ea95c3fa-7b40-411c-b59e-145971751437\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " Jul 28 16:40:04.568624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls\") pod \"ea95c3fa-7b40-411c-b59e-145971751437\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " Jul 28 16:40:04.568624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568468 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location\") pod \"ea95c3fa-7b40-411c-b59e-145971751437\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " Jul 28 16:40:04.568624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568487 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config\") pod \"ea95c3fa-7b40-411c-b59e-145971751437\" (UID: \"ea95c3fa-7b40-411c-b59e-145971751437\") " Jul 28 16:40:04.568804 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea95c3fa-7b40-411c-b59e-145971751437" (UID: "ea95c3fa-7b40-411c-b59e-145971751437"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:40:04.568859 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568842 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea95c3fa-7b40-411c-b59e-145971751437" (UID: "ea95c3fa-7b40-411c-b59e-145971751437"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:40:04.568917 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.568900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea95c3fa-7b40-411c-b59e-145971751437" (UID: "ea95c3fa-7b40-411c-b59e-145971751437"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:40:04.570556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.570533 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94" (OuterVolumeSpecName: "kube-api-access-lss94") pod "ea95c3fa-7b40-411c-b59e-145971751437" (UID: "ea95c3fa-7b40-411c-b59e-145971751437"). InnerVolumeSpecName "kube-api-access-lss94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:40:04.570680 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.570613 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea95c3fa-7b40-411c-b59e-145971751437" (UID: "ea95c3fa-7b40-411c-b59e-145971751437"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:40:04.669880 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.669844 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea95c3fa-7b40-411c-b59e-145971751437-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:40:04.669880 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.669880 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea95c3fa-7b40-411c-b59e-145971751437-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:40:04.670112 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.669895 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:40:04.670112 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.669911 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lss94\" (UniqueName: \"kubernetes.io/projected/ea95c3fa-7b40-411c-b59e-145971751437-kube-api-access-lss94\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:40:04.670112 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.669925 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea95c3fa-7b40-411c-b59e-145971751437-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:40:04.805087 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.805055 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:40:04.808899 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:04.808875 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2tjcd"] Jul 28 16:40:06.375871 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:40:06.375839 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea95c3fa-7b40-411c-b59e-145971751437" path="/var/lib/kubelet/pods/ea95c3fa-7b40-411c-b59e-145971751437/volumes" Jul 28 16:48:32.008359 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008326 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008616 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="storage-initializer" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008629 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="storage-initializer" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008641 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008648 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008661 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kube-rbac-proxy" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008669 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kube-rbac-proxy" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008716 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kserve-container" Jul 28 16:48:32.008799 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.008723 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea95c3fa-7b40-411c-b59e-145971751437" containerName="kube-rbac-proxy" Jul 28 16:48:32.011700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.011684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.013514 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.013483 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:48:32.013657 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.013627 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:48:32.013657 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.013633 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:48:32.013785 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.013642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\"" Jul 28 16:48:32.014240 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.014222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:48:32.014345 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.014221 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\"" Jul 28 16:48:32.014345 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.014287 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:48:32.021435 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.021412 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:48:32.101243 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.101213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.101434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.101253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.101434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.101365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.101434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.101409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjql\" (UniqueName: \"kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.101548 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.101440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202690 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.202654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.202711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.202731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjql\" (UniqueName: \"kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.202758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.202793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.202856 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:48:32.202829 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found Jul 28 16:48:32.203090 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:48:32.202923 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls podName:99f5b119-3248-4b3a-8f85-6e2f387ba7bf nodeName:}" failed. No retries permitted until 2026-07-28 16:48:32.70289299 +0000 UTC m=+4530.903740462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found Jul 28 16:48:32.203287 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.203263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.203473 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.203455 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.203515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.203462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.212998 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.212973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjql\" (UniqueName: \"kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.707001 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.706965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.709376 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.709352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-w49h4\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:32.922972 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:32.922933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:33.055717 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:33.055679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:48:33.058776 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:48:33.058746 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f5b119_3248_4b3a_8f85_6e2f387ba7bf.slice/crio-389ef75739e026958a369eb5e20976dae319f8eb90b2688586c304866c9a5ee7 WatchSource:0}: Error finding container 389ef75739e026958a369eb5e20976dae319f8eb90b2688586c304866c9a5ee7: Status 404 returned error can't find the container with id 389ef75739e026958a369eb5e20976dae319f8eb90b2688586c304866c9a5ee7 Jul 28 16:48:33.220220 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:33.220180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerStarted","Data":"c49cf5960caeeeee50229a685ca4c93409bf3570ca3059b01eeb4ae2b13acb54"} Jul 28 16:48:33.220379 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:33.220255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerStarted","Data":"389ef75739e026958a369eb5e20976dae319f8eb90b2688586c304866c9a5ee7"} Jul 28 16:48:38.239312 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:38.239271 2575 generic.go:356] "Generic (PLEG): container finished" podID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerID="c49cf5960caeeeee50229a685ca4c93409bf3570ca3059b01eeb4ae2b13acb54" exitCode=0 Jul 28 16:48:38.239691 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:38.239343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerDied","Data":"c49cf5960caeeeee50229a685ca4c93409bf3570ca3059b01eeb4ae2b13acb54"} Jul 28 16:48:38.240544 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:38.240530 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 16:48:42.257199 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:42.257160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerStarted","Data":"7734a00fc936d3b36fa99e3c7247197d6eee66cf84b937be89d1aeb00e9f9885"} Jul 28 16:48:42.257643 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:42.257253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerStarted","Data":"cc500fcc58b903acad04e3e117779141bc49e649baa566307761d891710b5726"} Jul 28 16:48:42.257643 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:42.257432 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:42.277567 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:42.277273 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podStartSLOduration=3.015121711 podStartE2EDuration="11.277252736s" totalImagesPullingTime="3.784201364s" totalInitContainerRuntime="4.477929661s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:48:42.276672894 +0000 UTC m=+4540.477520384" watchObservedRunningTime="2026-07-28 16:48:42.277252736 +0000 UTC m=+4540.478100205" Jul 28 16:48:43.260456 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:43.260428 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:43.261742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:43.261715 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 28 16:48:44.263989 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:44.263949 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 28 16:48:49.269041 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:49.269008 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:48:49.269628 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:49.269596 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 28 16:48:59.270543 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:48:59.270515 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:49:13.349327 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:13.349286 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:49:13.349787 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:13.349583 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" containerID="cri-o://cc500fcc58b903acad04e3e117779141bc49e649baa566307761d891710b5726" gracePeriod=30 Jul 28 16:49:13.349787 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:13.349655 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" containerID="cri-o://7734a00fc936d3b36fa99e3c7247197d6eee66cf84b937be89d1aeb00e9f9885" gracePeriod=30 Jul 28 16:49:14.265238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:14.265190 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:14.374792 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:14.374760 2575 generic.go:356] "Generic (PLEG): container finished" podID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerID="7734a00fc936d3b36fa99e3c7247197d6eee66cf84b937be89d1aeb00e9f9885" exitCode=2 Jul 28 16:49:14.375156 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:14.374827 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerDied","Data":"7734a00fc936d3b36fa99e3c7247197d6eee66cf84b937be89d1aeb00e9f9885"} Jul 28 16:49:18.537788 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.537752 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:49:18.540918 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.540903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.543110 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.543089 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\"" Jul 28 16:49:18.543225 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.543165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\"" Jul 28 16:49:18.552863 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.552839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:49:18.568825 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.568795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq44\" (UniqueName: \"kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.568955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.568839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.568955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.568883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.568955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.568909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.568955 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.568930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.669735 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.669698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq44\" (UniqueName: \"kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.669916 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.669755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.669916 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.669780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.669916 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.669810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.669916 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.669838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.670238 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.670216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.670574 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.670553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.670624 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.670588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.672257 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.672236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.679434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.679366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq44\" (UniqueName: \"kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44\") pod \"isvc-triton-predictor-5d7db58676-6pb5s\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.850923 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.850816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:49:18.979174 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:18.979139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:49:18.982132 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:49:18.982100 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619e336e_f573_4e93_8a79_b1765ba59399.slice/crio-a8423f347e6c202eac3024e410ff74c1b43d51cf4125e581f433e43c97f15c3c WatchSource:0}: Error finding container a8423f347e6c202eac3024e410ff74c1b43d51cf4125e581f433e43c97f15c3c: Status 404 returned error can't find the container with id a8423f347e6c202eac3024e410ff74c1b43d51cf4125e581f433e43c97f15c3c Jul 28 16:49:19.264617 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:19.264580 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:19.393332 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:19.393300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerStarted","Data":"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0"} Jul 28 16:49:19.393517 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:19.393358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerStarted","Data":"a8423f347e6c202eac3024e410ff74c1b43d51cf4125e581f433e43c97f15c3c"} Jul 28 16:49:23.409834 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:23.409712 2575 generic.go:356] "Generic (PLEG): container finished" podID="619e336e-f573-4e93-8a79-b1765ba59399" containerID="875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0" exitCode=0 Jul 28 16:49:23.409834 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:23.409792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerDied","Data":"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0"} Jul 28 16:49:24.264986 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:24.264945 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:24.265187 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:24.265080 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:49:29.264906 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:29.264553 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:34.264890 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:34.264385 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:39.264753 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:39.264699 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 28 16:49:43.526444 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:43.526406 2575 generic.go:356] "Generic (PLEG): container finished" podID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerID="cc500fcc58b903acad04e3e117779141bc49e649baa566307761d891710b5726" exitCode=137 Jul 28 16:49:43.526920 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:43.526495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerDied","Data":"cc500fcc58b903acad04e3e117779141bc49e649baa566307761d891710b5726"} Jul 28 16:49:44.062924 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.062821 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:49:44.190695 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.190608 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") pod \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " Jul 28 16:49:44.190695 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.190678 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddjql\" (UniqueName: \"kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql\") pod \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " Jul 28 16:49:44.190952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.190734 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location\") pod \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " Jul 28 16:49:44.190952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.190816 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert\") pod \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " Jul 28 16:49:44.190952 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.190849 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config\") pod \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\" (UID: \"99f5b119-3248-4b3a-8f85-6e2f387ba7bf\") " Jul 28 16:49:44.191481 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.191452 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99f5b119-3248-4b3a-8f85-6e2f387ba7bf" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:49:44.191602 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.191572 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99f5b119-3248-4b3a-8f85-6e2f387ba7bf" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:49:44.195192 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.194292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql" (OuterVolumeSpecName: "kube-api-access-ddjql") pod "99f5b119-3248-4b3a-8f85-6e2f387ba7bf" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf"). InnerVolumeSpecName "kube-api-access-ddjql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:49:44.195192 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.194737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99f5b119-3248-4b3a-8f85-6e2f387ba7bf" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:49:44.203925 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.203893 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99f5b119-3248-4b3a-8f85-6e2f387ba7bf" (UID: "99f5b119-3248-4b3a-8f85-6e2f387ba7bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:49:44.292427 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.292367 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:49:44.292427 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.292423 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:49:44.292667 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.292440 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:49:44.292667 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.292455 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddjql\" (UniqueName: \"kubernetes.io/projected/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kube-api-access-ddjql\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:49:44.292667 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.292469 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f5b119-3248-4b3a-8f85-6e2f387ba7bf-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:49:44.533843 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.533804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" event={"ID":"99f5b119-3248-4b3a-8f85-6e2f387ba7bf","Type":"ContainerDied","Data":"389ef75739e026958a369eb5e20976dae319f8eb90b2688586c304866c9a5ee7"} Jul 28 16:49:44.534315 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.533928 2575 scope.go:117] "RemoveContainer" containerID="7734a00fc936d3b36fa99e3c7247197d6eee66cf84b937be89d1aeb00e9f9885" Jul 28 16:49:44.534315 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.533980 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4" Jul 28 16:49:44.545494 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.545471 2575 scope.go:117] "RemoveContainer" containerID="cc500fcc58b903acad04e3e117779141bc49e649baa566307761d891710b5726" Jul 28 16:49:44.554999 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.554938 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:49:44.556239 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.556094 2575 scope.go:117] "RemoveContainer" containerID="c49cf5960caeeeee50229a685ca4c93409bf3570ca3059b01eeb4ae2b13acb54" Jul 28 16:49:44.559207 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:44.559146 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-w49h4"] Jul 28 16:49:46.375432 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:49:46.375382 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" path="/var/lib/kubelet/pods/99f5b119-3248-4b3a-8f85-6e2f387ba7bf/volumes" Jul 28 16:51:16.917496 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.917448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerStarted","Data":"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e"} Jul 28 16:51:16.918024 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.917569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerStarted","Data":"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac"} Jul 28 16:51:16.918024 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.917609 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:16.918024 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.917651 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:16.918959 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.918930 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Jul 28 16:51:16.948136 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:16.948077 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podStartSLOduration=1.840140544 podStartE2EDuration="1m58.948044046s" totalImagesPullingTime="1m53.075608876s" totalInitContainerRuntime="4.032294626s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:51:16.947465407 +0000 UTC m=+4695.148312898" watchObservedRunningTime="2026-07-28 16:51:16.948044046 +0000 UTC m=+4695.148891538" Jul 28 16:51:17.921213 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:17.921170 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Jul 28 16:51:22.925929 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:22.925897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:22.926378 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:22.926359 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Jul 28 16:51:32.927374 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:32.927342 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:40.002898 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:40.002859 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:51:40.003443 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:40.003124 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" containerID="cri-o://b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac" gracePeriod=30 Jul 28 16:51:40.003443 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:40.003210 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kube-rbac-proxy" containerID="cri-o://fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e" gracePeriod=30 Jul 28 16:51:41.000160 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:41.000121 2575 generic.go:356] "Generic (PLEG): container finished" podID="619e336e-f573-4e93-8a79-b1765ba59399" containerID="fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e" exitCode=2 Jul 28 16:51:41.000338 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:41.000190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerDied","Data":"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e"} Jul 28 16:51:42.963699 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:42.963669 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:43.009315 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.009278 2575 generic.go:356] "Generic (PLEG): container finished" podID="619e336e-f573-4e93-8a79-b1765ba59399" containerID="b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac" exitCode=0 Jul 28 16:51:43.009496 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.009331 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" Jul 28 16:51:43.009496 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.009345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerDied","Data":"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac"} Jul 28 16:51:43.009496 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.009420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" event={"ID":"619e336e-f573-4e93-8a79-b1765ba59399","Type":"ContainerDied","Data":"a8423f347e6c202eac3024e410ff74c1b43d51cf4125e581f433e43c97f15c3c"} Jul 28 16:51:43.009496 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.009451 2575 scope.go:117] "RemoveContainer" containerID="fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e" Jul 28 16:51:43.019197 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.018617 2575 scope.go:117] "RemoveContainer" containerID="b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac" Jul 28 16:51:43.027008 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.026987 2575 scope.go:117] "RemoveContainer" containerID="875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0" Jul 28 16:51:43.034217 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.034199 2575 scope.go:117] "RemoveContainer" containerID="fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e" Jul 28 16:51:43.034488 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:51:43.034468 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e\": container with ID starting with fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e not found: ID does not exist" containerID="fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e" Jul 28 16:51:43.034584 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.034503 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e"} err="failed to get container status \"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e\": rpc error: code = NotFound desc = could not find container \"fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e\": container with ID starting with fa340b407c90ad9bb11c26558be0b3cb33dd83013f46e4e0c2ca1c57cb73469e not found: ID does not exist" Jul 28 16:51:43.034584 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.034529 2575 scope.go:117] "RemoveContainer" containerID="b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac" Jul 28 16:51:43.034813 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:51:43.034795 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac\": container with ID starting with b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac not found: ID does not exist" containerID="b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac" Jul 28 16:51:43.034856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.034823 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac"} err="failed to get container status \"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac\": rpc error: code = NotFound desc = could not find container \"b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac\": container with ID starting with b6bcc3c2436e44db4a7787b7fe22570fae065b9a358c2a536081455d3ddf9eac not found: ID does not exist" Jul 28 16:51:43.034856 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.034839 2575 scope.go:117] "RemoveContainer" containerID="875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0" Jul 28 16:51:43.035075 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:51:43.035057 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0\": container with ID starting with 875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0 not found: ID does not exist" containerID="875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0" Jul 28 16:51:43.035120 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.035080 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0"} err="failed to get container status \"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0\": rpc error: code = NotFound desc = could not find container \"875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0\": container with ID starting with 875be3ed7ceee57b57ca4b4a5055f5adc1425f8c17e4d316af71d2c5bb3ad5a0 not found: ID does not exist" Jul 28 16:51:43.092484 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092372 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config\") pod \"619e336e-f573-4e93-8a79-b1765ba59399\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " Jul 28 16:51:43.092484 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location\") pod \"619e336e-f573-4e93-8a79-b1765ba59399\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " Jul 28 16:51:43.092679 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092513 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert\") pod \"619e336e-f573-4e93-8a79-b1765ba59399\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " Jul 28 16:51:43.092679 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls\") pod \"619e336e-f573-4e93-8a79-b1765ba59399\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " Jul 28 16:51:43.092679 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gq44\" (UniqueName: \"kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44\") pod \"619e336e-f573-4e93-8a79-b1765ba59399\" (UID: \"619e336e-f573-4e93-8a79-b1765ba59399\") " Jul 28 16:51:43.092901 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092863 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "619e336e-f573-4e93-8a79-b1765ba59399" (UID: "619e336e-f573-4e93-8a79-b1765ba59399"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:51:43.092949 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "619e336e-f573-4e93-8a79-b1765ba59399" (UID: "619e336e-f573-4e93-8a79-b1765ba59399"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:51:43.092949 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.092916 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "619e336e-f573-4e93-8a79-b1765ba59399" (UID: "619e336e-f573-4e93-8a79-b1765ba59399"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:51:43.094755 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.094733 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "619e336e-f573-4e93-8a79-b1765ba59399" (UID: "619e336e-f573-4e93-8a79-b1765ba59399"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:51:43.094862 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.094843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44" (OuterVolumeSpecName: "kube-api-access-5gq44") pod "619e336e-f573-4e93-8a79-b1765ba59399" (UID: "619e336e-f573-4e93-8a79-b1765ba59399"). InnerVolumeSpecName "kube-api-access-5gq44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:51:43.193749 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.193714 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/619e336e-f573-4e93-8a79-b1765ba59399-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:51:43.193749 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.193744 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:51:43.193749 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.193755 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/619e336e-f573-4e93-8a79-b1765ba59399-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:51:43.194058 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.193763 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gq44\" (UniqueName: \"kubernetes.io/projected/619e336e-f573-4e93-8a79-b1765ba59399-kube-api-access-5gq44\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:51:43.194058 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.193772 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/619e336e-f573-4e93-8a79-b1765ba59399-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:51:43.334943 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.334906 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:51:43.340487 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.340458 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s"] Jul 28 16:51:43.922281 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.922229 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": context deadline exceeded" Jul 28 16:51:43.927257 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:43.927231 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-6pb5s" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: i/o timeout" Jul 28 16:51:44.371704 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:44.371672 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619e336e-f573-4e93-8a79-b1765ba59399" path="/var/lib/kubelet/pods/619e336e-f573-4e93-8a79-b1765ba59399/volumes" Jul 28 16:51:45.163407 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163356 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:51:45.163649 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163636 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163650 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163661 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163667 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163679 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="storage-initializer" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163685 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="storage-initializer" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163691 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" Jul 28 16:51:45.163700 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163695 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163708 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="storage-initializer" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163713 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="storage-initializer" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163718 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kube-rbac-proxy" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163723 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kube-rbac-proxy" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163761 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kube-rbac-proxy" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163768 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kube-rbac-proxy" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163775 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="619e336e-f573-4e93-8a79-b1765ba59399" containerName="kserve-container" Jul 28 16:51:45.163911 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.163781 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f5b119-3248-4b3a-8f85-6e2f387ba7bf" containerName="kserve-container" Jul 28 16:51:45.199046 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.199010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:51:45.199205 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.199149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.201361 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201328 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:51:45.201361 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:51:45.201556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201369 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\"" Jul 28 16:51:45.201556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201370 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:51:45.201556 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:51:45.201956 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.201939 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:51:45.202016 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.202002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\"" Jul 28 16:51:45.311528 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.311491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4vz\" (UniqueName: \"kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.311528 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.311526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.311786 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.311555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.311786 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.311670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.311786 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.311727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.412462 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.412413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4vz\" (UniqueName: \"kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.412462 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.412464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.412987 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.412502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.412987 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.412548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.412987 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.412597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.413242 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.413057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.413340 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.413314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.413542 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.413493 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.415132 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.415115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.421752 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.421727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4vz\" (UniqueName: \"kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz\") pod \"isvc-xgboost-predictor-7566d65bf7-nfblx\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.509406 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.509361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:51:45.664047 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:45.663963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:51:45.667288 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:51:45.667252 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda090d2ed_492f_4c40_bba5_4e7b02e1d8e1.slice/crio-c18d45b9d80f1b6c8aebebf238ef59e43b079105ad633f1eccc69256fd73676d WatchSource:0}: Error finding container c18d45b9d80f1b6c8aebebf238ef59e43b079105ad633f1eccc69256fd73676d: Status 404 returned error can't find the container with id c18d45b9d80f1b6c8aebebf238ef59e43b079105ad633f1eccc69256fd73676d Jul 28 16:51:46.022483 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:46.022446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerStarted","Data":"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b"} Jul 28 16:51:46.022689 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:46.022505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerStarted","Data":"c18d45b9d80f1b6c8aebebf238ef59e43b079105ad633f1eccc69256fd73676d"} Jul 28 16:51:50.038874 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:50.038837 2575 generic.go:356] "Generic (PLEG): container finished" podID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerID="720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b" exitCode=0 Jul 28 16:51:50.039267 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:51:50.038897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerDied","Data":"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b"} Jul 28 16:52:09.113377 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:09.113344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerStarted","Data":"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4"} Jul 28 16:52:09.114710 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:09.113421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerStarted","Data":"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072"} Jul 28 16:52:09.114710 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:09.113696 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:52:09.133696 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:09.133642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podStartSLOduration=1.596526208 podStartE2EDuration="24.133625758s" totalImagesPullingTime="18.630524276s" totalInitContainerRuntime="3.906575274s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:52:09.131979929 +0000 UTC m=+4747.332827420" watchObservedRunningTime="2026-07-28 16:52:09.133625758 +0000 UTC m=+4747.334473249" Jul 28 16:52:10.117303 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:10.117277 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:52:10.118671 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:10.118642 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:11.121067 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:11.121022 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:16.125977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:16.125945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:52:16.126629 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:16.126595 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:26.126906 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:26.126806 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:36.127103 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:36.127062 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:46.127076 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:46.127035 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:52:56.127133 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:52:56.127086 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:53:06.126515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:06.126470 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:53:16.127969 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:16.127934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:53:25.302508 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:25.302470 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:53:25.302994 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:25.302711 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" containerID="cri-o://7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072" gracePeriod=30 Jul 28 16:53:25.302994 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:25.302751 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kube-rbac-proxy" containerID="cri-o://ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4" gracePeriod=30 Jul 28 16:53:26.121654 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:26.121605 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused" Jul 28 16:53:26.126897 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:26.126853 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 16:53:26.381703 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:26.381615 2575 generic.go:356] "Generic (PLEG): container finished" podID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerID="ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4" exitCode=2 Jul 28 16:53:26.381703 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:26.381660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerDied","Data":"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4"} Jul 28 16:53:28.946822 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:28.946800 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:53:29.023070 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023032 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location\") pod \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " Jul 28 16:53:29.023245 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls\") pod \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " Jul 28 16:53:29.023245 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023105 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert\") pod \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " Jul 28 16:53:29.023245 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023127 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt4vz\" (UniqueName: \"kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz\") pod \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " Jul 28 16:53:29.023245 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023183 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config\") pod \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\" (UID: \"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1\") " Jul 28 16:53:29.023515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" (UID: "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:53:29.023564 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" (UID: "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:53:29.023620 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.023601 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" (UID: "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:53:29.025289 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.025259 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" (UID: "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:53:29.025386 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.025315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz" (OuterVolumeSpecName: "kube-api-access-nt4vz") pod "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" (UID: "a090d2ed-492f-4c40-bba5-4e7b02e1d8e1"). InnerVolumeSpecName "kube-api-access-nt4vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:53:29.124571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.124479 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:53:29.124571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.124513 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:53:29.124571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.124523 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4vz\" (UniqueName: \"kubernetes.io/projected/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-api-access-nt4vz\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:53:29.124571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.124531 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:53:29.124571 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.124540 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:53:29.393444 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.393329 2575 generic.go:356] "Generic (PLEG): container finished" podID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerID="7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072" exitCode=0 Jul 28 16:53:29.393444 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.393411 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" Jul 28 16:53:29.393444 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.393416 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerDied","Data":"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072"} Jul 28 16:53:29.393663 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.393475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx" event={"ID":"a090d2ed-492f-4c40-bba5-4e7b02e1d8e1","Type":"ContainerDied","Data":"c18d45b9d80f1b6c8aebebf238ef59e43b079105ad633f1eccc69256fd73676d"} Jul 28 16:53:29.393663 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.393501 2575 scope.go:117] "RemoveContainer" containerID="ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4" Jul 28 16:53:29.401868 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.401848 2575 scope.go:117] "RemoveContainer" containerID="7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072" Jul 28 16:53:29.410906 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.410885 2575 scope.go:117] "RemoveContainer" containerID="720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b" Jul 28 16:53:29.418032 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418014 2575 scope.go:117] "RemoveContainer" containerID="ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4" Jul 28 16:53:29.418322 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:53:29.418299 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4\": container with ID starting with ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4 not found: ID does not exist" containerID="ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4" Jul 28 16:53:29.418473 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418330 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4"} err="failed to get container status \"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4\": rpc error: code = NotFound desc = could not find container \"ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4\": container with ID starting with ec3f0a2301d6b169cd14bc93a4b6c06d3b314d5d0fc0bd7ed8574e4aa89bbad4 not found: ID does not exist" Jul 28 16:53:29.418473 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418349 2575 scope.go:117] "RemoveContainer" containerID="7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072" Jul 28 16:53:29.418615 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:53:29.418598 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072\": container with ID starting with 7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072 not found: ID does not exist" containerID="7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072" Jul 28 16:53:29.418658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418621 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072"} err="failed to get container status \"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072\": rpc error: code = NotFound desc = could not find container \"7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072\": container with ID starting with 7fb3a6b330aab189b9ea1c377f40d5780f21c2836ef3cdfa972eb7a5b9ca3072 not found: ID does not exist" Jul 28 16:53:29.418658 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418636 2575 scope.go:117] "RemoveContainer" containerID="720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b" Jul 28 16:53:29.418841 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:53:29.418825 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b\": container with ID starting with 720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b not found: ID does not exist" containerID="720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b" Jul 28 16:53:29.418886 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.418846 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b"} err="failed to get container status \"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b\": rpc error: code = NotFound desc = could not find container \"720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b\": container with ID starting with 720d8577f1c0badc4760de3dae8a5fab47c5e981fa3ac7f0c5e9c50eec0fcc6b not found: ID does not exist" Jul 28 16:53:29.423059 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.423035 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:53:29.441142 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:29.441101 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nfblx"] Jul 28 16:53:30.371877 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:53:30.371844 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" path="/var/lib/kubelet/pods/a090d2ed-492f-4c40-bba5-4e7b02e1d8e1/volumes" Jul 28 16:55:20.982868 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.982831 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983125 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="storage-initializer" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983139 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="storage-initializer" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983165 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kube-rbac-proxy" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983172 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kube-rbac-proxy" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983183 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983193 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983238 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kube-rbac-proxy" Jul 28 16:55:20.983373 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.983246 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a090d2ed-492f-4c40-bba5-4e7b02e1d8e1" containerName="kserve-container" Jul 28 16:55:20.985204 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.985182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:20.987442 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987420 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:55:20.987608 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\"" Jul 28 16:55:20.987706 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987638 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\"" Jul 28 16:55:20.987706 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:55:20.987706 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987665 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:55:20.987871 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.987588 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:55:20.988101 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.988086 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:55:20.996228 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:20.996193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:55:21.043615 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.043580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.043795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.043625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.043795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.043647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.043795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.043687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.043795 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.043782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qx92\" (UniqueName: \"kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144483 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qx92\" (UniqueName: \"kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144675 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144675 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144769 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144769 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.144874 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:55:21.144844 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found Jul 28 16:55:21.144957 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:55:21.144942 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls podName:23911ed4-8867-44ca-ba23-afe2d4686398 nodeName:}" failed. No retries permitted until 2026-07-28 16:55:21.644920935 +0000 UTC m=+4939.845768419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-p6t75" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found Jul 28 16:55:21.145028 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.144982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.145338 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.145318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.145447 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.145430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.155726 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.155701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qx92\" (UniqueName: \"kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.648507 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.648467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.650879 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.650851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p6t75\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:21.896361 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:21.896304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:22.028419 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:22.028370 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:55:22.031717 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:55:22.031679 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23911ed4_8867_44ca_ba23_afe2d4686398.slice/crio-955599b929a48abfb1639b8e369579f13da73d833b737b79b15b0dc4a6022b3f WatchSource:0}: Error finding container 955599b929a48abfb1639b8e369579f13da73d833b737b79b15b0dc4a6022b3f: Status 404 returned error can't find the container with id 955599b929a48abfb1639b8e369579f13da73d833b737b79b15b0dc4a6022b3f Jul 28 16:55:22.783075 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:22.783038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerStarted","Data":"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c"} Jul 28 16:55:22.783252 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:22.783097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerStarted","Data":"955599b929a48abfb1639b8e369579f13da73d833b737b79b15b0dc4a6022b3f"} Jul 28 16:55:26.796257 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:26.796215 2575 generic.go:356] "Generic (PLEG): container finished" podID="23911ed4-8867-44ca-ba23-afe2d4686398" containerID="df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c" exitCode=0 Jul 28 16:55:26.796257 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:26.796263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerDied","Data":"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c"} Jul 28 16:55:27.802524 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.802484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerStarted","Data":"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed"} Jul 28 16:55:27.802972 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.802573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerStarted","Data":"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a"} Jul 28 16:55:27.802972 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.802885 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:27.803081 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.803028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:27.804241 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.804217 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:55:27.822871 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:27.822795 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podStartSLOduration=3.743630486 podStartE2EDuration="7.822775669s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079145183s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:55:27.821825266 +0000 UTC m=+4946.022672736" watchObservedRunningTime="2026-07-28 16:55:27.822775669 +0000 UTC m=+4946.023623169" Jul 28 16:55:28.807113 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:28.807072 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:55:29.810102 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:29.810047 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:55:34.814486 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:34.814457 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:55:34.814963 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:34.814936 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:55:44.815616 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:44.815578 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:55:54.814988 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:55:54.814948 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:56:04.815552 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:04.815509 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:56:14.814891 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:14.814854 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:56:24.814928 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:24.814890 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 16:56:34.815540 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:34.815511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:56:41.092568 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:41.092528 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:56:41.092937 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:41.092873 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" containerID="cri-o://1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a" gracePeriod=30 Jul 28 16:56:41.093004 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:41.092940 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kube-rbac-proxy" containerID="cri-o://805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed" gracePeriod=30 Jul 28 16:56:42.062001 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:42.061966 2575 generic.go:356] "Generic (PLEG): container finished" podID="23911ed4-8867-44ca-ba23-afe2d4686398" containerID="805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed" exitCode=2 Jul 28 16:56:42.062177 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:42.062037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerDied","Data":"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed"} Jul 28 16:56:44.747052 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.747026 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:56:44.803428 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803385 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert\") pod \"23911ed4-8867-44ca-ba23-afe2d4686398\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " Jul 28 16:56:44.803632 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803498 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") pod \"23911ed4-8867-44ca-ba23-afe2d4686398\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " Jul 28 16:56:44.803632 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803532 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location\") pod \"23911ed4-8867-44ca-ba23-afe2d4686398\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " Jul 28 16:56:44.803632 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config\") pod \"23911ed4-8867-44ca-ba23-afe2d4686398\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " Jul 28 16:56:44.803632 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803592 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qx92\" (UniqueName: \"kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92\") pod \"23911ed4-8867-44ca-ba23-afe2d4686398\" (UID: \"23911ed4-8867-44ca-ba23-afe2d4686398\") " Jul 28 16:56:44.803887 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23911ed4-8867-44ca-ba23-afe2d4686398" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:56:44.803956 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803905 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23911ed4-8867-44ca-ba23-afe2d4686398" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:56:44.804010 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.803981 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23911ed4-8867-44ca-ba23-afe2d4686398" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:56:44.805811 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.805783 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23911ed4-8867-44ca-ba23-afe2d4686398" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:56:44.805914 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.805843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92" (OuterVolumeSpecName: "kube-api-access-4qx92") pod "23911ed4-8867-44ca-ba23-afe2d4686398" (UID: "23911ed4-8867-44ca-ba23-afe2d4686398"). InnerVolumeSpecName "kube-api-access-4qx92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:56:44.904669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.904622 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:56:44.904669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.904666 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23911ed4-8867-44ca-ba23-afe2d4686398-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:56:44.904669 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.904676 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23911ed4-8867-44ca-ba23-afe2d4686398-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:56:44.904897 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.904686 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23911ed4-8867-44ca-ba23-afe2d4686398-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:56:44.904897 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:44.904695 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qx92\" (UniqueName: \"kubernetes.io/projected/23911ed4-8867-44ca-ba23-afe2d4686398-kube-api-access-4qx92\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:56:45.075325 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.075289 2575 generic.go:356] "Generic (PLEG): container finished" podID="23911ed4-8867-44ca-ba23-afe2d4686398" containerID="1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a" exitCode=0 Jul 28 16:56:45.075542 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.075342 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" Jul 28 16:56:45.075542 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.075351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerDied","Data":"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a"} Jul 28 16:56:45.075542 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.075437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75" event={"ID":"23911ed4-8867-44ca-ba23-afe2d4686398","Type":"ContainerDied","Data":"955599b929a48abfb1639b8e369579f13da73d833b737b79b15b0dc4a6022b3f"} Jul 28 16:56:45.075542 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.075475 2575 scope.go:117] "RemoveContainer" containerID="805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed" Jul 28 16:56:45.083522 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.083500 2575 scope.go:117] "RemoveContainer" containerID="1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a" Jul 28 16:56:45.090585 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.090571 2575 scope.go:117] "RemoveContainer" containerID="df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c" Jul 28 16:56:45.096002 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.095982 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:56:45.098226 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098051 2575 scope.go:117] "RemoveContainer" containerID="805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed" Jul 28 16:56:45.098385 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:56:45.098347 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed\": container with ID starting with 805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed not found: ID does not exist" containerID="805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed" Jul 28 16:56:45.098538 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098452 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed"} err="failed to get container status \"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed\": rpc error: code = NotFound desc = could not find container \"805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed\": container with ID starting with 805eedc904d8167faba66bbd0f6208c662fcf0e4797c60add953cd904d8992ed not found: ID does not exist" Jul 28 16:56:45.098538 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098478 2575 scope.go:117] "RemoveContainer" containerID="1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a" Jul 28 16:56:45.098722 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098705 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p6t75"] Jul 28 16:56:45.098771 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:56:45.098737 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a\": container with ID starting with 1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a not found: ID does not exist" containerID="1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a" Jul 28 16:56:45.098771 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098757 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a"} err="failed to get container status \"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a\": rpc error: code = NotFound desc = could not find container \"1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a\": container with ID starting with 1d93347ef2572e2aba8e35200e085b55ceb3b6211cf46e8a023498bec9ce5f5a not found: ID does not exist" Jul 28 16:56:45.098843 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.098780 2575 scope.go:117] "RemoveContainer" containerID="df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c" Jul 28 16:56:45.099021 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:56:45.099003 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c\": container with ID starting with df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c not found: ID does not exist" containerID="df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c" Jul 28 16:56:45.099078 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:45.099028 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c"} err="failed to get container status \"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c\": rpc error: code = NotFound desc = could not find container \"df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c\": container with ID starting with df079ef50badef0fcc1e03f297738ac589782683920e253d6f68177408fd393c not found: ID does not exist" Jul 28 16:56:46.372236 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:56:46.372203 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" path="/var/lib/kubelet/pods/23911ed4-8867-44ca-ba23-afe2d4686398/volumes" Jul 28 16:57:51.569174 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569134 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569512 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569528 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569540 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kube-rbac-proxy" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569549 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kube-rbac-proxy" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569562 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="storage-initializer" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569571 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="storage-initializer" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569633 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kserve-container" Jul 28 16:57:51.569665 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.569645 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23911ed4-8867-44ca-ba23-afe2d4686398" containerName="kube-rbac-proxy" Jul 28 16:57:51.572930 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.572911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.575574 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.575546 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\"" Jul 28 16:57:51.575769 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.575550 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:57:51.575769 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.575557 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:57:51.575769 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.575561 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\"" Jul 28 16:57:51.575997 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.575980 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:57:51.576073 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.576001 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:57:51.576120 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.576103 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:57:51.584532 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.584505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:57:51.622038 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.622003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.622216 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.622044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.622216 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.622118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.622216 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.622201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvkx9\" (UniqueName: \"kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.622321 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.622278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722709 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.722676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722709 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.722714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.722735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.722758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722977 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.722799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvkx9\" (UniqueName: \"kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.722977 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:57:51.722906 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found Jul 28 16:57:51.723201 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:57:51.722988 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls podName:d9c8f781-7e82-4c68-9d5c-2522172a5afe nodeName:}" failed. No retries permitted until 2026-07-28 16:57:52.222967248 +0000 UTC m=+5090.423814721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-9dh6k" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe") : secret "isvc-xgboost-v2-predictor-serving-cert" not found Jul 28 16:57:51.723201 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.723152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.723545 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.723525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.723639 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.723620 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:51.734056 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:51.734027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvkx9\" (UniqueName: \"kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:52.226435 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:52.226372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:52.228802 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:52.228782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-9dh6k\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:52.483625 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:52.483534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:52.615081 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:52.615035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:57:52.617969 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:57:52.617935 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c8f781_7e82_4c68_9d5c_2522172a5afe.slice/crio-5084724d8541999e9594603a6e3ca9c4f30e813fe419eaec180997ef688e654a WatchSource:0}: Error finding container 5084724d8541999e9594603a6e3ca9c4f30e813fe419eaec180997ef688e654a: Status 404 returned error can't find the container with id 5084724d8541999e9594603a6e3ca9c4f30e813fe419eaec180997ef688e654a Jul 28 16:57:53.321998 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:53.321962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerStarted","Data":"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7"} Jul 28 16:57:53.322183 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:53.322017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerStarted","Data":"5084724d8541999e9594603a6e3ca9c4f30e813fe419eaec180997ef688e654a"} Jul 28 16:57:57.337673 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:57.337636 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerID="16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7" exitCode=0 Jul 28 16:57:57.338053 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:57.337713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerDied","Data":"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7"} Jul 28 16:57:58.344034 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.343997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerStarted","Data":"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a"} Jul 28 16:57:58.344507 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.344058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerStarted","Data":"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6"} Jul 28 16:57:58.344507 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.344406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:58.344596 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.344537 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:57:58.345912 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.345886 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:57:58.364958 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:58.364909 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podStartSLOduration=3.644390648 podStartE2EDuration="7.364892713s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.720502065s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:57:58.364632059 +0000 UTC m=+5096.565479551" watchObservedRunningTime="2026-07-28 16:57:58.364892713 +0000 UTC m=+5096.565740204" Jul 28 16:57:59.348524 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:57:59.348485 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:00.352581 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:00.352542 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:05.357934 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:05.357902 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:58:05.358540 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:05.358512 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:15.359518 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:15.359479 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:25.358985 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:25.358897 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:35.358837 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:35.358792 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:45.358638 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:45.358597 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:58:55.359163 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:58:55.359120 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 16:59:05.359544 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:05.359513 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:59:11.679713 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:11.679682 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:59:11.680145 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:11.679923 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" containerID="cri-o://3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6" gracePeriod=30 Jul 28 16:59:11.680145 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:11.679963 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kube-rbac-proxy" containerID="cri-o://b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a" gracePeriod=30 Jul 28 16:59:12.609933 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:12.609901 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerID="b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a" exitCode=2 Jul 28 16:59:12.610106 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:12.609968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerDied","Data":"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a"} Jul 28 16:59:15.420002 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.419977 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:59:15.436829 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.436798 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location\") pod \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " Jul 28 16:59:15.437000 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.436838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert\") pod \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " Jul 28 16:59:15.437000 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.436873 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config\") pod \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " Jul 28 16:59:15.437000 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.436909 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvkx9\" (UniqueName: \"kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9\") pod \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " Jul 28 16:59:15.437173 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") pod \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\" (UID: \"d9c8f781-7e82-4c68-9d5c-2522172a5afe\") " Jul 28 16:59:15.437173 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437153 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9c8f781-7e82-4c68-9d5c-2522172a5afe" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:59:15.437276 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437215 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9c8f781-7e82-4c68-9d5c-2522172a5afe" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:59:15.437276 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9c8f781-7e82-4c68-9d5c-2522172a5afe" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:59:15.437355 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437325 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:59:15.437355 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437351 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:59:15.437457 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.437366 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:59:15.439067 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.439044 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9" (OuterVolumeSpecName: "kube-api-access-nvkx9") pod "d9c8f781-7e82-4c68-9d5c-2522172a5afe" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe"). InnerVolumeSpecName "kube-api-access-nvkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:59:15.439127 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.439052 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9c8f781-7e82-4c68-9d5c-2522172a5afe" (UID: "d9c8f781-7e82-4c68-9d5c-2522172a5afe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:59:15.538728 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.538691 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c8f781-7e82-4c68-9d5c-2522172a5afe-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:59:15.538728 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.538727 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvkx9\" (UniqueName: \"kubernetes.io/projected/d9c8f781-7e82-4c68-9d5c-2522172a5afe-kube-api-access-nvkx9\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 16:59:15.623650 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.623615 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerID="3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6" exitCode=0 Jul 28 16:59:15.623851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.623675 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" Jul 28 16:59:15.623851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.623705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerDied","Data":"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6"} Jul 28 16:59:15.623851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.623768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" event={"ID":"d9c8f781-7e82-4c68-9d5c-2522172a5afe","Type":"ContainerDied","Data":"5084724d8541999e9594603a6e3ca9c4f30e813fe419eaec180997ef688e654a"} Jul 28 16:59:15.623851 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.623797 2575 scope.go:117] "RemoveContainer" containerID="b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a" Jul 28 16:59:15.632709 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.632688 2575 scope.go:117] "RemoveContainer" containerID="3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6" Jul 28 16:59:15.639832 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.639815 2575 scope.go:117] "RemoveContainer" containerID="16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7" Jul 28 16:59:15.645165 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.645141 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:59:15.647237 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.647223 2575 scope.go:117] "RemoveContainer" containerID="b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a" Jul 28 16:59:15.647554 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:59:15.647531 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a\": container with ID starting with b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a not found: ID does not exist" containerID="b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a" Jul 28 16:59:15.647657 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.647565 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a"} err="failed to get container status \"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a\": rpc error: code = NotFound desc = could not find container \"b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a\": container with ID starting with b3555bfd9218b1d844ccd7a94ce9ecee9cde78fe3c2e8fe510e101bfe7b2ba1a not found: ID does not exist" Jul 28 16:59:15.647657 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.647591 2575 scope.go:117] "RemoveContainer" containerID="3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6" Jul 28 16:59:15.647881 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:59:15.647858 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6\": container with ID starting with 3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6 not found: ID does not exist" containerID="3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6" Jul 28 16:59:15.648000 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.647890 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6"} err="failed to get container status \"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6\": rpc error: code = NotFound desc = could not find container \"3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6\": container with ID starting with 3a0a13be69aec22939eebfdf2f44dcf3720d02fee69f9182ca200749c1a335d6 not found: ID does not exist" Jul 28 16:59:15.648000 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.647912 2575 scope.go:117] "RemoveContainer" containerID="16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7" Jul 28 16:59:15.648515 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:59:15.648479 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7\": container with ID starting with 16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7 not found: ID does not exist" containerID="16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7" Jul 28 16:59:15.648515 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.648510 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7"} err="failed to get container status \"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7\": rpc error: code = NotFound desc = could not find container \"16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7\": container with ID starting with 16683f77b408688854754bc292815f202ad7cb6597b6a4c794e2356d8d57aff7 not found: ID does not exist" Jul 28 16:59:15.649062 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:15.649043 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k"] Jul 28 16:59:16.353689 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.353640 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 16:59:16.359363 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.359334 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-9dh6k" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: i/o timeout" Jul 28 16:59:16.371648 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.371622 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" path="/var/lib/kubelet/pods/d9c8f781-7e82-4c68-9d5c-2522172a5afe/volumes" Jul 28 16:59:16.858332 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858293 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858652 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="storage-initializer" Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858667 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="storage-initializer" Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858680 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858697 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kube-rbac-proxy" Jul 28 16:59:16.858745 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858707 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kube-rbac-proxy" Jul 28 16:59:16.859103 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858776 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kserve-container" Jul 28 16:59:16.859103 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.858786 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8f781-7e82-4c68-9d5c-2522172a5afe" containerName="kube-rbac-proxy" Jul 28 16:59:16.863486 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.863459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:16.865575 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.865552 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\"" Jul 28 16:59:16.865575 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.865562 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:59:16.865789 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.865565 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Jul 28 16:59:16.866304 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.866283 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\"" Jul 28 16:59:16.866477 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.866307 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 16:59:16.866477 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.866325 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:59:16.872463 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.872437 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 16:59:16.949541 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.949503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:16.949734 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.949553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:16.949734 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.949636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnzv\" (UniqueName: \"kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:16.949734 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:16.949707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050194 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnzv\" (UniqueName: \"kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050410 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050410 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050410 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050587 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:59:17.050424 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found Jul 28 16:59:17.050587 ip-10-0-136-8 kubenswrapper[2575]: E0728 16:59:17.050496 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls podName:42f94932-b743-4ef1-8261-d8ed3a770bcf nodeName:}" failed. No retries permitted until 2026-07-28 16:59:17.550474889 +0000 UTC m=+5175.751322372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls") pod "isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" (UID: "42f94932-b743-4ef1-8261-d8ed3a770bcf") : secret "isvc-sklearn-s3-predictor-serving-cert" not found Jul 28 16:59:17.050739 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.050950 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.050922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.062434 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.062409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnzv\" (UniqueName: \"kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.553068 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.553024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.555442 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.555409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-j5dq7\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.775011 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.774968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:17.907141 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:17.907119 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 16:59:17.909791 ip-10-0-136-8 kubenswrapper[2575]: W0728 16:59:17.909758 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f94932_b743_4ef1_8261_d8ed3a770bcf.slice/crio-ec4180c4fd9ab9a37d1d57ebac019dae3d7b842bd1d02d715f7797f47de4e1ce WatchSource:0}: Error finding container ec4180c4fd9ab9a37d1d57ebac019dae3d7b842bd1d02d715f7797f47de4e1ce: Status 404 returned error can't find the container with id ec4180c4fd9ab9a37d1d57ebac019dae3d7b842bd1d02d715f7797f47de4e1ce Jul 28 16:59:18.637538 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:18.637500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerStarted","Data":"af69c3f66f5723105d20160a3455909c1fd7c7931f215aabb8cf2481ebb515e6"} Jul 28 16:59:18.637742 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:18.637558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerStarted","Data":"ec4180c4fd9ab9a37d1d57ebac019dae3d7b842bd1d02d715f7797f47de4e1ce"} Jul 28 16:59:19.645093 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:19.645059 2575 generic.go:356] "Generic (PLEG): container finished" podID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerID="af69c3f66f5723105d20160a3455909c1fd7c7931f215aabb8cf2481ebb515e6" exitCode=0 Jul 28 16:59:19.645594 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:19.645116 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerDied","Data":"af69c3f66f5723105d20160a3455909c1fd7c7931f215aabb8cf2481ebb515e6"} Jul 28 16:59:20.650903 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.650866 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerStarted","Data":"41b2ebdd9db39cb4c930ab9921953e2cb0277547f1cfbfb4a1cade925e6f3558"} Jul 28 16:59:20.651421 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.650923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerStarted","Data":"39bd1886cf35281163fcc55d25bc7395c3c391b8d777f336663570a337eac958"} Jul 28 16:59:20.651421 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.651126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:20.651421 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.651233 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:20.652597 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.652573 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 16:59:20.674036 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:20.673981 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podStartSLOduration=3.640108831 podStartE2EDuration="4.673962895s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.033854064s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:59:20.673597547 +0000 UTC m=+5178.874445050" watchObservedRunningTime="2026-07-28 16:59:20.673962895 +0000 UTC m=+5178.874810389" Jul 28 16:59:21.654544 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:21.654503 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 16:59:26.660256 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:26.660226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 16:59:26.660748 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:26.660723 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 16:59:36.661671 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:36.661625 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 16:59:46.661016 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:46.660975 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 16:59:56.661519 ip-10-0-136-8 kubenswrapper[2575]: I0728 16:59:56.661434 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 17:00:06.661383 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:06.661341 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 17:00:16.661702 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:16.661659 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 17:00:26.661559 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:26.661517 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Jul 28 17:00:33.368562 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:33.368522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 17:00:37.029097 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.029070 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 17:00:37.029536 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.029280 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" containerID="cri-o://39bd1886cf35281163fcc55d25bc7395c3c391b8d777f336663570a337eac958" gracePeriod=30 Jul 28 17:00:37.029536 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.029318 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kube-rbac-proxy" containerID="cri-o://41b2ebdd9db39cb4c930ab9921953e2cb0277547f1cfbfb4a1cade925e6f3558" gracePeriod=30 Jul 28 17:00:37.156871 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.156845 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:00:37.159967 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.159950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.161869 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.161843 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\"" Jul 28 17:00:37.161969 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.161918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:00:37.162155 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.162132 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\"" Jul 28 17:00:37.171481 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.171456 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:00:37.271379 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.271336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.271379 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.271381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.271657 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.271480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4kj\" (UniqueName: \"kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.271657 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.271521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.271657 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.271540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372245 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372245 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372245 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4kj\" (UniqueName: \"kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372245 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372608 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372608 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:00:37.372369 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found Jul 28 17:00:37.372608 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:00:37.372485 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls podName:88a15176-2a19-4842-8ba1-1e5e1127a4e7 nodeName:}" failed. No retries permitted until 2026-07-28 17:00:37.872460884 +0000 UTC m=+5256.073308354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found Jul 28 17:00:37.372788 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372677 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372916 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.372973 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.372957 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.382410 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.382371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4kj\" (UniqueName: \"kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.876437 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.876380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.878809 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.878781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:37.927822 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.927793 2575 generic.go:356] "Generic (PLEG): container finished" podID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerID="41b2ebdd9db39cb4c930ab9921953e2cb0277547f1cfbfb4a1cade925e6f3558" exitCode=2 Jul 28 17:00:37.927940 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:37.927868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerDied","Data":"41b2ebdd9db39cb4c930ab9921953e2cb0277547f1cfbfb4a1cade925e6f3558"} Jul 28 17:00:38.070134 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:38.070091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:38.204873 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:38.204841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:00:38.208170 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:00:38.208144 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a15176_2a19_4842_8ba1_1e5e1127a4e7.slice/crio-1b8b8ad95e103f75fb7323f1f852f2dd048371faba64a1c5c0b82719cea434e3 WatchSource:0}: Error finding container 1b8b8ad95e103f75fb7323f1f852f2dd048371faba64a1c5c0b82719cea434e3: Status 404 returned error can't find the container with id 1b8b8ad95e103f75fb7323f1f852f2dd048371faba64a1c5c0b82719cea434e3 Jul 28 17:00:38.933070 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:38.933031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerStarted","Data":"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5"} Jul 28 17:00:38.933245 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:38.933140 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerStarted","Data":"1b8b8ad95e103f75fb7323f1f852f2dd048371faba64a1c5c0b82719cea434e3"} Jul 28 17:00:39.937720 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:39.937684 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerID="8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5" exitCode=0 Jul 28 17:00:39.938309 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:39.937729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerDied","Data":"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5"} Jul 28 17:00:40.944000 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.943958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerStarted","Data":"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8"} Jul 28 17:00:40.944430 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.944016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerStarted","Data":"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb"} Jul 28 17:00:40.944430 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.944040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:40.944430 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.944061 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:40.945543 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.945516 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:00:40.963628 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:40.963576 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podStartSLOduration=2.875186108 podStartE2EDuration="3.963560172s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.088374064s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:00:40.962343293 +0000 UTC m=+5259.163190784" watchObservedRunningTime="2026-07-28 17:00:40.963560172 +0000 UTC m=+5259.164407663" Jul 28 17:00:41.654805 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:41.654761 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused" Jul 28 17:00:41.951486 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:41.951382 2575 generic.go:356] "Generic (PLEG): container finished" podID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerID="39bd1886cf35281163fcc55d25bc7395c3c391b8d777f336663570a337eac958" exitCode=0 Jul 28 17:00:41.951486 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:41.951425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerDied","Data":"39bd1886cf35281163fcc55d25bc7395c3c391b8d777f336663570a337eac958"} Jul 28 17:00:41.951874 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:41.951815 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:00:41.974155 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:41.974132 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 17:00:42.012984 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.012953 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location\") pod \"42f94932-b743-4ef1-8261-d8ed3a770bcf\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " Jul 28 17:00:42.013143 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config\") pod \"42f94932-b743-4ef1-8261-d8ed3a770bcf\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " Jul 28 17:00:42.013143 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013047 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") pod \"42f94932-b743-4ef1-8261-d8ed3a770bcf\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " Jul 28 17:00:42.013143 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hnzv\" (UniqueName: \"kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv\") pod \"42f94932-b743-4ef1-8261-d8ed3a770bcf\" (UID: \"42f94932-b743-4ef1-8261-d8ed3a770bcf\") " Jul 28 17:00:42.013415 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42f94932-b743-4ef1-8261-d8ed3a770bcf" (UID: "42f94932-b743-4ef1-8261-d8ed3a770bcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:00:42.013415 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013351 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42f94932-b743-4ef1-8261-d8ed3a770bcf" (UID: "42f94932-b743-4ef1-8261-d8ed3a770bcf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:00:42.013415 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.013412 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f94932-b743-4ef1-8261-d8ed3a770bcf-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:00:42.015758 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.015732 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42f94932-b743-4ef1-8261-d8ed3a770bcf" (UID: "42f94932-b743-4ef1-8261-d8ed3a770bcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:00:42.015856 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.015747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv" (OuterVolumeSpecName: "kube-api-access-6hnzv") pod "42f94932-b743-4ef1-8261-d8ed3a770bcf" (UID: "42f94932-b743-4ef1-8261-d8ed3a770bcf"). InnerVolumeSpecName "kube-api-access-6hnzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:00:42.114159 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.114124 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnzv\" (UniqueName: \"kubernetes.io/projected/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-api-access-6hnzv\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:00:42.114159 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.114151 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f94932-b743-4ef1-8261-d8ed3a770bcf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:00:42.114159 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.114161 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f94932-b743-4ef1-8261-d8ed3a770bcf-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:00:42.958279 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.958250 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" Jul 28 17:00:42.958279 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.958281 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7" event={"ID":"42f94932-b743-4ef1-8261-d8ed3a770bcf","Type":"ContainerDied","Data":"ec4180c4fd9ab9a37d1d57ebac019dae3d7b842bd1d02d715f7797f47de4e1ce"} Jul 28 17:00:42.958772 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.958357 2575 scope.go:117] "RemoveContainer" containerID="41b2ebdd9db39cb4c930ab9921953e2cb0277547f1cfbfb4a1cade925e6f3558" Jul 28 17:00:42.967003 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.966832 2575 scope.go:117] "RemoveContainer" containerID="39bd1886cf35281163fcc55d25bc7395c3c391b8d777f336663570a337eac958" Jul 28 17:00:42.976779 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.976750 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 17:00:42.977789 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.977769 2575 scope.go:117] "RemoveContainer" containerID="af69c3f66f5723105d20160a3455909c1fd7c7931f215aabb8cf2481ebb515e6" Jul 28 17:00:42.980760 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:42.980737 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-j5dq7"] Jul 28 17:00:44.371801 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:44.371757 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" path="/var/lib/kubelet/pods/42f94932-b743-4ef1-8261-d8ed3a770bcf/volumes" Jul 28 17:00:46.956220 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:46.956186 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:00:46.956800 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:46.956769 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:00:56.956852 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:00:56.956808 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:06.956821 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:06.956779 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:16.957312 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:16.957273 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:26.956969 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:26.956886 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:36.957637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:36.957594 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:46.957264 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:46.957228 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Jul 28 17:01:56.957589 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:01:56.957557 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:02:07.208601 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:07.208567 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:02:07.209125 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:07.208871 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" containerID="cri-o://3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb" gracePeriod=30 Jul 28 17:02:07.209125 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:07.208887 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kube-rbac-proxy" containerID="cri-o://e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8" gracePeriod=30 Jul 28 17:02:08.267226 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.267192 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerID="e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8" exitCode=2 Jul 28 17:02:08.267636 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.267242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerDied","Data":"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8"} Jul 28 17:02:08.349930 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.349900 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:08.350200 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350186 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kube-rbac-proxy" Jul 28 17:02:08.350244 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350203 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kube-rbac-proxy" Jul 28 17:02:08.350244 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350214 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="storage-initializer" Jul 28 17:02:08.350244 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350225 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="storage-initializer" Jul 28 17:02:08.350244 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350236 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" Jul 28 17:02:08.350244 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350242 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" Jul 28 17:02:08.350415 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350299 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kube-rbac-proxy" Jul 28 17:02:08.350415 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.350309 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f94932-b743-4ef1-8261-d8ed3a770bcf" containerName="kserve-container" Jul 28 17:02:08.353152 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.353133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.354927 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.354888 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\"" Jul 28 17:02:08.355024 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.354967 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\"" Jul 28 17:02:08.364005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.363980 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:08.480729 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.480689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.480912 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.480747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.480912 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.480820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdg2\" (UniqueName: \"kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.480912 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.480883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.581646 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.581544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.581646 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.581604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.581646 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.581631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdg2\" (UniqueName: \"kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.581941 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.581683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.581941 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:02:08.581816 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found Jul 28 17:02:08.581941 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:02:08.581893 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls podName:1f9c4fef-32ec-4537-947d-7a4a76d5015e nodeName:}" failed. No retries permitted until 2026-07-28 17:02:09.081872035 +0000 UTC m=+5347.282719507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" (UID: "1f9c4fef-32ec-4537-947d-7a4a76d5015e") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found Jul 28 17:02:08.582096 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.581989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.582262 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.582241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:08.590376 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:08.590350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdg2\" (UniqueName: \"kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:09.087097 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:09.087047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:09.089496 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:09.089462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:09.262978 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:09.262932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:09.396897 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:09.396665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:09.399385 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:02:09.399360 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9c4fef_32ec_4537_947d_7a4a76d5015e.slice/crio-0b159547d66367db3fccddeff70f572a9ade0e939b9ebccc2a9d41ad105b67a8 WatchSource:0}: Error finding container 0b159547d66367db3fccddeff70f572a9ade0e939b9ebccc2a9d41ad105b67a8: Status 404 returned error can't find the container with id 0b159547d66367db3fccddeff70f572a9ade0e939b9ebccc2a9d41ad105b67a8 Jul 28 17:02:10.276554 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:10.276517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerStarted","Data":"c3dbbab1fb30fc36a1ebca5dfc596aad467c489aadf31caf2c5289de9ce1befb"} Jul 28 17:02:10.276732 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:10.276572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerStarted","Data":"0b159547d66367db3fccddeff70f572a9ade0e939b9ebccc2a9d41ad105b67a8"} Jul 28 17:02:11.953001 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:11.952956 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused" Jul 28 17:02:12.162440 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.162410 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:02:12.290599 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.290560 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerID="3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb" exitCode=0 Jul 28 17:02:12.290771 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.290624 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" Jul 28 17:02:12.290771 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.290636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerDied","Data":"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb"} Jul 28 17:02:12.290771 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.290727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x" event={"ID":"88a15176-2a19-4842-8ba1-1e5e1127a4e7","Type":"ContainerDied","Data":"1b8b8ad95e103f75fb7323f1f852f2dd048371faba64a1c5c0b82719cea434e3"} Jul 28 17:02:12.290771 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.290768 2575 scope.go:117] "RemoveContainer" containerID="e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8" Jul 28 17:02:12.292251 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.292232 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/0.log" Jul 28 17:02:12.292331 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.292264 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerID="c3dbbab1fb30fc36a1ebca5dfc596aad467c489aadf31caf2c5289de9ce1befb" exitCode=1 Jul 28 17:02:12.292331 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.292319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerDied","Data":"c3dbbab1fb30fc36a1ebca5dfc596aad467c489aadf31caf2c5289de9ce1befb"} Jul 28 17:02:12.299528 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.299510 2575 scope.go:117] "RemoveContainer" containerID="3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb" Jul 28 17:02:12.307359 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.307338 2575 scope.go:117] "RemoveContainer" containerID="8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5" Jul 28 17:02:12.313064 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp4kj\" (UniqueName: \"kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj\") pod \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " Jul 28 17:02:12.313182 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config\") pod \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " Jul 28 17:02:12.313182 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313126 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") pod \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " Jul 28 17:02:12.313182 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert\") pod \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " Jul 28 17:02:12.313408 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location\") pod \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\" (UID: \"88a15176-2a19-4842-8ba1-1e5e1127a4e7\") " Jul 28 17:02:12.313631 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313603 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88a15176-2a19-4842-8ba1-1e5e1127a4e7" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:02:12.313713 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88a15176-2a19-4842-8ba1-1e5e1127a4e7" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:02:12.313853 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.313803 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88a15176-2a19-4842-8ba1-1e5e1127a4e7" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:02:12.315510 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.315488 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88a15176-2a19-4842-8ba1-1e5e1127a4e7" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:02:12.315664 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.315650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj" (OuterVolumeSpecName: "kube-api-access-zp4kj") pod "88a15176-2a19-4842-8ba1-1e5e1127a4e7" (UID: "88a15176-2a19-4842-8ba1-1e5e1127a4e7"). InnerVolumeSpecName "kube-api-access-zp4kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:02:12.316298 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.316284 2575 scope.go:117] "RemoveContainer" containerID="e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8" Jul 28 17:02:12.316600 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:02:12.316583 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8\": container with ID starting with e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8 not found: ID does not exist" containerID="e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8" Jul 28 17:02:12.316665 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.316606 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8"} err="failed to get container status \"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8\": rpc error: code = NotFound desc = could not find container \"e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8\": container with ID starting with e985f8e7947a75a01eb411a7b03acce2a0c00589d7419ceb583cdf3edf7457a8 not found: ID does not exist" Jul 28 17:02:12.316665 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.316623 2575 scope.go:117] "RemoveContainer" containerID="3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb" Jul 28 17:02:12.316896 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:02:12.316878 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb\": container with ID starting with 3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb not found: ID does not exist" containerID="3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb" Jul 28 17:02:12.316944 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.316902 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb"} err="failed to get container status \"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb\": rpc error: code = NotFound desc = could not find container \"3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb\": container with ID starting with 3400b2fb05b43f871665d77e4a0c1822d86b9dc2321a6c8aaa4b3d381c60c6eb not found: ID does not exist" Jul 28 17:02:12.316944 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.316918 2575 scope.go:117] "RemoveContainer" containerID="8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5" Jul 28 17:02:12.317136 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:02:12.317117 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5\": container with ID starting with 8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5 not found: ID does not exist" containerID="8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5" Jul 28 17:02:12.317178 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.317139 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5"} err="failed to get container status \"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5\": rpc error: code = NotFound desc = could not find container \"8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5\": container with ID starting with 8614fa3029df36042a9ddbdeccbefe85a134f74015134e09b79e61d9908fa0b5 not found: ID does not exist" Jul 28 17:02:12.414558 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.414519 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88a15176-2a19-4842-8ba1-1e5e1127a4e7-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:12.414558 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.414550 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:12.414809 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.414629 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:12.414809 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.414775 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4kj\" (UniqueName: \"kubernetes.io/projected/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-api-access-zp4kj\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:12.414809 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.414803 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88a15176-2a19-4842-8ba1-1e5e1127a4e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:12.606777 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.606694 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:02:12.611827 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:12.611795 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-dvl4x"] Jul 28 17:02:13.298122 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:13.298095 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/0.log" Jul 28 17:02:13.298581 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:13.298211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerStarted","Data":"69c92636cc40e4040d00cf00b614b7ce4fead3c25b7e49cf9d7fde939f1d667a"} Jul 28 17:02:14.371865 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:14.371828 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" path="/var/lib/kubelet/pods/88a15176-2a19-4842-8ba1-1e5e1127a4e7/volumes" Jul 28 17:02:18.311400 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.311345 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:18.311798 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.311610 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" containerID="cri-o://69c92636cc40e4040d00cf00b614b7ce4fead3c25b7e49cf9d7fde939f1d667a" gracePeriod=30 Jul 28 17:02:18.319073 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.319054 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/1.log" Jul 28 17:02:18.319507 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.319493 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/0.log" Jul 28 17:02:18.319572 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.319524 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerID="69c92636cc40e4040d00cf00b614b7ce4fead3c25b7e49cf9d7fde939f1d667a" exitCode=1 Jul 28 17:02:18.319608 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.319588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerDied","Data":"69c92636cc40e4040d00cf00b614b7ce4fead3c25b7e49cf9d7fde939f1d667a"} Jul 28 17:02:18.319654 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.319642 2575 scope.go:117] "RemoveContainer" containerID="c3dbbab1fb30fc36a1ebca5dfc596aad467c489aadf31caf2c5289de9ce1befb" Jul 28 17:02:18.450655 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.450634 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/1.log" Jul 28 17:02:18.454131 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.454113 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:18.561138 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561095 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location\") pod \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " Jul 28 17:02:18.561319 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config\") pod \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " Jul 28 17:02:18.561319 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zdg2\" (UniqueName: \"kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2\") pod \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " Jul 28 17:02:18.561319 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561242 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") pod \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\" (UID: \"1f9c4fef-32ec-4537-947d-7a4a76d5015e\") " Jul 28 17:02:18.561555 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561439 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f9c4fef-32ec-4537-947d-7a4a76d5015e" (UID: "1f9c4fef-32ec-4537-947d-7a4a76d5015e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:02:18.561555 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.561546 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f9c4fef-32ec-4537-947d-7a4a76d5015e" (UID: "1f9c4fef-32ec-4537-947d-7a4a76d5015e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:02:18.563320 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.563296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2" (OuterVolumeSpecName: "kube-api-access-8zdg2") pod "1f9c4fef-32ec-4537-947d-7a4a76d5015e" (UID: "1f9c4fef-32ec-4537-947d-7a4a76d5015e"). InnerVolumeSpecName "kube-api-access-8zdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:02:18.563385 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.563361 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f9c4fef-32ec-4537-947d-7a4a76d5015e" (UID: "1f9c4fef-32ec-4537-947d-7a4a76d5015e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:02:18.662237 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.662196 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:18.662237 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.662229 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:18.662237 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.662242 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdg2\" (UniqueName: \"kubernetes.io/projected/1f9c4fef-32ec-4537-947d-7a4a76d5015e-kube-api-access-8zdg2\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:18.662576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:18.662255 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9c4fef-32ec-4537-947d-7a4a76d5015e-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:02:19.324158 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.324131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb_1f9c4fef-32ec-4537-947d-7a4a76d5015e/storage-initializer/1.log" Jul 28 17:02:19.324583 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.324227 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" Jul 28 17:02:19.324583 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.324227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb" event={"ID":"1f9c4fef-32ec-4537-947d-7a4a76d5015e","Type":"ContainerDied","Data":"0b159547d66367db3fccddeff70f572a9ade0e939b9ebccc2a9d41ad105b67a8"} Jul 28 17:02:19.324583 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.324304 2575 scope.go:117] "RemoveContainer" containerID="69c92636cc40e4040d00cf00b614b7ce4fead3c25b7e49cf9d7fde939f1d667a" Jul 28 17:02:19.367447 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.367412 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:19.372490 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.372457 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-5s6tb"] Jul 28 17:02:19.500709 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500674 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:02:19.500945 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500931 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500960 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500967 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500973 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500978 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500986 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.500990 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="storage-initializer" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501003 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kube-rbac-proxy" Jul 28 17:02:19.501005 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501008 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kube-rbac-proxy" Jul 28 17:02:19.501467 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501046 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501467 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501053 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kserve-container" Jul 28 17:02:19.501467 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501060 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" containerName="storage-initializer" Jul 28 17:02:19.501467 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.501065 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a15176-2a19-4842-8ba1-1e5e1127a4e7" containerName="kube-rbac-proxy" Jul 28 17:02:19.505509 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.505490 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.507991 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.507967 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\"" Jul 28 17:02:19.508200 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\"" Jul 28 17:02:19.508462 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508431 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:02:19.508561 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:02:19.508561 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508538 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:02:19.508847 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508828 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 17:02:19.508938 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.508900 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Jul 28 17:02:19.514838 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.514814 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:02:19.670685 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.670582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.670685 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.670624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.670932 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.670710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.670932 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.670792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.670932 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.670819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4p9\" (UniqueName: \"kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771381 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771381 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771658 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771658 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771658 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4p9\" (UniqueName: \"kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.771973 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.771936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.772115 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.772065 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.772187 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.772142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.773875 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.773851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.780791 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.780765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4p9\" (UniqueName: \"kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.817597 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.817557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:19.946750 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:19.946675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:02:19.949542 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:02:19.949512 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc605ab54_4e56_4f9c_b278_5ce7d6618a65.slice/crio-056ff25eccb1eabcbc58b587fe13f52b98e1d432cc7eb39f0499d973a909ffb2 WatchSource:0}: Error finding container 056ff25eccb1eabcbc58b587fe13f52b98e1d432cc7eb39f0499d973a909ffb2: Status 404 returned error can't find the container with id 056ff25eccb1eabcbc58b587fe13f52b98e1d432cc7eb39f0499d973a909ffb2 Jul 28 17:02:20.333142 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:20.333102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerStarted","Data":"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e"} Jul 28 17:02:20.333648 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:20.333181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerStarted","Data":"056ff25eccb1eabcbc58b587fe13f52b98e1d432cc7eb39f0499d973a909ffb2"} Jul 28 17:02:20.374954 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:20.374921 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f9c4fef-32ec-4537-947d-7a4a76d5015e" path="/var/lib/kubelet/pods/1f9c4fef-32ec-4537-947d-7a4a76d5015e/volumes" Jul 28 17:02:21.338990 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:21.338954 2575 generic.go:356] "Generic (PLEG): container finished" podID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerID="dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e" exitCode=0 Jul 28 17:02:21.339484 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:21.339029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerDied","Data":"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e"} Jul 28 17:02:22.344643 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:22.344609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerStarted","Data":"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b"} Jul 28 17:02:22.345026 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:22.344695 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerStarted","Data":"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62"} Jul 28 17:02:22.345026 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:22.344810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:22.365737 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:22.365692 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podStartSLOduration=2.277082058 podStartE2EDuration="3.365676917s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.088594859s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:02:22.364628716 +0000 UTC m=+5360.565476206" watchObservedRunningTime="2026-07-28 17:02:22.365676917 +0000 UTC m=+5360.566524407" Jul 28 17:02:23.348271 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:23.348240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:23.349460 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:23.349434 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:02:24.351757 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:24.351719 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:02:29.356526 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:29.356495 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:02:29.356965 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:29.356929 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:02:39.356954 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:39.356914 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:02:49.357173 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:49.357131 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:02:59.357226 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:02:59.357126 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:03:09.357984 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:09.357938 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:03:19.357370 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:19.357331 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:03:29.357866 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:29.357821 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Jul 28 17:03:39.358437 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:39.358387 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:03:49.589382 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:49.589347 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:03:49.589795 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:49.589637 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" containerID="cri-o://83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62" gracePeriod=30 Jul 28 17:03:49.589795 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:49.589653 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kube-rbac-proxy" containerID="cri-o://8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b" gracePeriod=30 Jul 28 17:03:50.662141 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.662107 2575 generic.go:356] "Generic (PLEG): container finished" podID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerID="8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b" exitCode=2 Jul 28 17:03:50.662622 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.662169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerDied","Data":"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b"} Jul 28 17:03:50.664789 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.664765 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:03:50.668029 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.668015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.669996 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.669973 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\"" Jul 28 17:03:50.670174 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.670157 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\"" Jul 28 17:03:50.682019 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.681993 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:03:50.725255 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.725219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.725255 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.725255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.725502 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.725282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sds\" (UniqueName: \"kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.725502 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.725354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826120 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826312 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826312 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826449 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sds\" (UniqueName: \"kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826699 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.826914 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.826887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.828596 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.828580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.836016 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.835990 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sds\" (UniqueName: \"kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:50.979080 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:50.978982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:03:51.109327 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:51.109294 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:03:51.113241 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:03:51.113215 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f8150_1b00_42c4_b1c8_2613462afb88.slice/crio-58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3 WatchSource:0}: Error finding container 58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3: Status 404 returned error can't find the container with id 58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3 Jul 28 17:03:51.667061 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:51.667006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerStarted","Data":"4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db"} Jul 28 17:03:51.667476 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:51.667101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerStarted","Data":"58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3"} Jul 28 17:03:54.351965 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.351919 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused" Jul 28 17:03:54.646994 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.646966 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:03:54.680102 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.680072 2575 generic.go:356] "Generic (PLEG): container finished" podID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerID="83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62" exitCode=0 Jul 28 17:03:54.680285 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.680148 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" Jul 28 17:03:54.680285 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.680162 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerDied","Data":"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62"} Jul 28 17:03:54.680285 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.680223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk" event={"ID":"c605ab54-4e56-4f9c-b278-5ce7d6618a65","Type":"ContainerDied","Data":"056ff25eccb1eabcbc58b587fe13f52b98e1d432cc7eb39f0499d973a909ffb2"} Jul 28 17:03:54.680285 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.680257 2575 scope.go:117] "RemoveContainer" containerID="8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b" Jul 28 17:03:54.681796 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.681777 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/0.log" Jul 28 17:03:54.681904 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.681809 2575 generic.go:356] "Generic (PLEG): container finished" podID="373f8150-1b00-42c4-b1c8-2613462afb88" containerID="4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db" exitCode=1 Jul 28 17:03:54.681904 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.681837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerDied","Data":"4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db"} Jul 28 17:03:54.688797 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.688778 2575 scope.go:117] "RemoveContainer" containerID="83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62" Jul 28 17:03:54.697289 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.697262 2575 scope.go:117] "RemoveContainer" containerID="dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e" Jul 28 17:03:54.716327 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.716302 2575 scope.go:117] "RemoveContainer" containerID="8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b" Jul 28 17:03:54.716691 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:03:54.716667 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b\": container with ID starting with 8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b not found: ID does not exist" containerID="8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b" Jul 28 17:03:54.716799 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.716699 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b"} err="failed to get container status \"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b\": rpc error: code = NotFound desc = could not find container \"8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b\": container with ID starting with 8e8092a2ecd1d5f8606d0beb8f394623d9cf3fb8a754fb40dd069d01b8a10d8b not found: ID does not exist" Jul 28 17:03:54.716799 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.716718 2575 scope.go:117] "RemoveContainer" containerID="83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62" Jul 28 17:03:54.717029 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:03:54.717006 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62\": container with ID starting with 83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62 not found: ID does not exist" containerID="83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62" Jul 28 17:03:54.717074 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.717033 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62"} err="failed to get container status \"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62\": rpc error: code = NotFound desc = could not find container \"83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62\": container with ID starting with 83264231f9403b6ac171ed45934c3b78600e30efa23a9973884b622edbf58a62 not found: ID does not exist" Jul 28 17:03:54.717074 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.717047 2575 scope.go:117] "RemoveContainer" containerID="dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e" Jul 28 17:03:54.717269 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:03:54.717249 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e\": container with ID starting with dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e not found: ID does not exist" containerID="dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e" Jul 28 17:03:54.717311 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.717273 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e"} err="failed to get container status \"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e\": rpc error: code = NotFound desc = could not find container \"dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e\": container with ID starting with dfc9f82fa4277e1136a77fc22038d9043114df791fff0269d785d547dabf245e not found: ID does not exist" Jul 28 17:03:54.759464 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759432 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert\") pod \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " Jul 28 17:03:54.759626 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759482 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls\") pod \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " Jul 28 17:03:54.759626 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759530 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config\") pod \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " Jul 28 17:03:54.759626 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location\") pod \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " Jul 28 17:03:54.759626 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd4p9\" (UniqueName: \"kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9\") pod \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\" (UID: \"c605ab54-4e56-4f9c-b278-5ce7d6618a65\") " Jul 28 17:03:54.759871 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c605ab54-4e56-4f9c-b278-5ce7d6618a65" (UID: "c605ab54-4e56-4f9c-b278-5ce7d6618a65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:03:54.759949 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c605ab54-4e56-4f9c-b278-5ce7d6618a65" (UID: "c605ab54-4e56-4f9c-b278-5ce7d6618a65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:03:54.760001 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.759956 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c605ab54-4e56-4f9c-b278-5ce7d6618a65" (UID: "c605ab54-4e56-4f9c-b278-5ce7d6618a65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:03:54.761745 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.761712 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c605ab54-4e56-4f9c-b278-5ce7d6618a65" (UID: "c605ab54-4e56-4f9c-b278-5ce7d6618a65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:03:54.761902 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.761885 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9" (OuterVolumeSpecName: "kube-api-access-wd4p9") pod "c605ab54-4e56-4f9c-b278-5ce7d6618a65" (UID: "c605ab54-4e56-4f9c-b278-5ce7d6618a65"). InnerVolumeSpecName "kube-api-access-wd4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:03:54.860406 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.860371 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:03:54.860406 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.860410 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4p9\" (UniqueName: \"kubernetes.io/projected/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-api-access-wd4p9\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:03:54.860613 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.860421 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:03:54.860613 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.860429 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c605ab54-4e56-4f9c-b278-5ce7d6618a65-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:03:54.860613 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:54.860436 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c605ab54-4e56-4f9c-b278-5ce7d6618a65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:03:55.003099 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:55.003067 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:03:55.006699 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:55.006670 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-htkpk"] Jul 28 17:03:55.687931 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:55.687899 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/0.log" Jul 28 17:03:55.688344 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:55.688016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerStarted","Data":"0dfb5ac8df83a515b8ef30b1492d0d05b79694105a5c9492f1064310eb631642"} Jul 28 17:03:56.372843 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.372811 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" path="/var/lib/kubelet/pods/c605ab54-4e56-4f9c-b278-5ce7d6618a65/volumes" Jul 28 17:03:56.694810 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.694734 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/1.log" Jul 28 17:03:56.695191 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.695070 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/0.log" Jul 28 17:03:56.695191 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.695103 2575 generic.go:356] "Generic (PLEG): container finished" podID="373f8150-1b00-42c4-b1c8-2613462afb88" containerID="0dfb5ac8df83a515b8ef30b1492d0d05b79694105a5c9492f1064310eb631642" exitCode=1 Jul 28 17:03:56.695191 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.695136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerDied","Data":"0dfb5ac8df83a515b8ef30b1492d0d05b79694105a5c9492f1064310eb631642"} Jul 28 17:03:56.695310 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.695199 2575 scope.go:117] "RemoveContainer" containerID="4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db" Jul 28 17:03:56.695569 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.695550 2575 scope.go:117] "RemoveContainer" containerID="4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db" Jul 28 17:03:56.706479 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:03:56.706453 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_kserve-ci-e2e-test(373f8150-1b00-42c4-b1c8-2613462afb88)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" Jul 28 17:03:56.706479 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:03:56.706457 2575 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_kserve-ci-e2e-test_373f8150-1b00-42c4-b1c8-2613462afb88_0 in pod sandbox 58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3 from index: no such id: '4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db'" containerID="4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db" Jul 28 17:03:56.706629 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:56.706496 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_kserve-ci-e2e-test_373f8150-1b00-42c4-b1c8-2613462afb88_0 in pod sandbox 58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3 from index: no such id: '4501cb5bbfc66eb6475317f32aadfe13c939bb44469405658184eada1e2915db'" Jul 28 17:03:57.700027 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:03:57.699995 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/1.log" Jul 28 17:04:00.649965 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.649937 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:04:00.778324 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.778302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/1.log" Jul 28 17:04:00.780559 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.780543 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:04:00.912706 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.912621 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config\") pod \"373f8150-1b00-42c4-b1c8-2613462afb88\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " Jul 28 17:04:00.912706 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.912683 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls\") pod \"373f8150-1b00-42c4-b1c8-2613462afb88\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " Jul 28 17:04:00.913014 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.912718 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57sds\" (UniqueName: \"kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds\") pod \"373f8150-1b00-42c4-b1c8-2613462afb88\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " Jul 28 17:04:00.913014 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.912767 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location\") pod \"373f8150-1b00-42c4-b1c8-2613462afb88\" (UID: \"373f8150-1b00-42c4-b1c8-2613462afb88\") " Jul 28 17:04:00.913139 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.913088 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "373f8150-1b00-42c4-b1c8-2613462afb88" (UID: "373f8150-1b00-42c4-b1c8-2613462afb88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:04:00.913190 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.913142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "373f8150-1b00-42c4-b1c8-2613462afb88" (UID: "373f8150-1b00-42c4-b1c8-2613462afb88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:04:00.914868 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.914837 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "373f8150-1b00-42c4-b1c8-2613462afb88" (UID: "373f8150-1b00-42c4-b1c8-2613462afb88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:04:00.914984 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:00.914880 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds" (OuterVolumeSpecName: "kube-api-access-57sds") pod "373f8150-1b00-42c4-b1c8-2613462afb88" (UID: "373f8150-1b00-42c4-b1c8-2613462afb88"). InnerVolumeSpecName "kube-api-access-57sds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:04:01.013456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.013417 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373f8150-1b00-42c4-b1c8-2613462afb88-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:04:01.013456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.013446 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57sds\" (UniqueName: \"kubernetes.io/projected/373f8150-1b00-42c4-b1c8-2613462afb88-kube-api-access-57sds\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:04:01.013456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.013456 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373f8150-1b00-42c4-b1c8-2613462afb88-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:04:01.013456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.013468 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373f8150-1b00-42c4-b1c8-2613462afb88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:04:01.717590 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.717561 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl_373f8150-1b00-42c4-b1c8-2613462afb88/storage-initializer/1.log" Jul 28 17:04:01.718009 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.717621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" event={"ID":"373f8150-1b00-42c4-b1c8-2613462afb88","Type":"ContainerDied","Data":"58a88a6d883aac1aea375f85ca511524063a847111aebf0d96a4d7e92ae4d1c3"} Jul 28 17:04:01.718009 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.717677 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl" Jul 28 17:04:01.718009 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.717684 2575 scope.go:117] "RemoveContainer" containerID="0dfb5ac8df83a515b8ef30b1492d0d05b79694105a5c9492f1064310eb631642" Jul 28 17:04:01.761381 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.761338 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:04:01.769622 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.769588 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-2rtbl"] Jul 28 17:04:01.789817 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.789782 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:04:01.790031 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790019 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kube-rbac-proxy" Jul 28 17:04:01.790071 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790032 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kube-rbac-proxy" Jul 28 17:04:01.790071 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790043 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.790071 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790049 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.790071 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790061 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" Jul 28 17:04:01.790071 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790067 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790075 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="storage-initializer" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790080 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="storage-initializer" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790086 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790091 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790134 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790141 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kserve-container" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790147 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c605ab54-4e56-4f9c-b278-5ce7d6618a65" containerName="kube-rbac-proxy" Jul 28 17:04:01.790224 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.790218 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" containerName="storage-initializer" Jul 28 17:04:01.794618 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.794598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:01.796918 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.796890 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\"" Jul 28 17:04:01.797047 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.796918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Jul 28 17:04:01.797165 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.797152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:04:01.797231 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.797213 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\"" Jul 28 17:04:01.797649 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.797634 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:04:01.797725 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.797656 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\"" Jul 28 17:04:01.797725 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.797700 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:04:01.804307 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.804282 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:04:01.919738 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.919693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:01.919738 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.919737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:01.919947 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.919759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz5p\" (UniqueName: \"kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:01.919947 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.919830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:01.919947 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:01.919886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.020985 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.020956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021138 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.020994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021138 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021138 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz5p\" (UniqueName: \"kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021287 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021454 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021767 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.021833 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.021750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.023901 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.023878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.030084 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.030061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz5p\" (UniqueName: \"kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.105190 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.105147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:02.240644 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.240599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:04:02.243571 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:04:02.243540 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode416d26b_b426_4308_b0f1_4fac932bd6cc.slice/crio-7453b78eb768ba4dd8ccaf0b3c7a22e24209858146887c3e2a895d8ef2345fa0 WatchSource:0}: Error finding container 7453b78eb768ba4dd8ccaf0b3c7a22e24209858146887c3e2a895d8ef2345fa0: Status 404 returned error can't find the container with id 7453b78eb768ba4dd8ccaf0b3c7a22e24209858146887c3e2a895d8ef2345fa0 Jul 28 17:04:02.248029 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.248006 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Jul 28 17:04:02.372692 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.372662 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373f8150-1b00-42c4-b1c8-2613462afb88" path="/var/lib/kubelet/pods/373f8150-1b00-42c4-b1c8-2613462afb88/volumes" Jul 28 17:04:02.723456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.723349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerStarted","Data":"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315"} Jul 28 17:04:02.723456 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:02.723439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerStarted","Data":"7453b78eb768ba4dd8ccaf0b3c7a22e24209858146887c3e2a895d8ef2345fa0"} Jul 28 17:04:03.729937 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:03.729843 2575 generic.go:356] "Generic (PLEG): container finished" podID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerID="e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315" exitCode=0 Jul 28 17:04:03.729937 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:03.729890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerDied","Data":"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315"} Jul 28 17:04:04.735849 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:04.735808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerStarted","Data":"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7"} Jul 28 17:04:04.736203 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:04.735881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerStarted","Data":"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f"} Jul 28 17:04:04.736203 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:04.735938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:04.756902 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:04.756849 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podStartSLOduration=2.647029315 podStartE2EDuration="3.756829979s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.109800664s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:04:04.755821317 +0000 UTC m=+5462.956668810" watchObservedRunningTime="2026-07-28 17:04:04.756829979 +0000 UTC m=+5462.957677471" Jul 28 17:04:05.740101 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:05.740068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:05.741205 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:05.741177 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:06.743174 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:06.743132 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:11.747361 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:11.747331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:04:11.747932 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:11.747906 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:21.748151 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:21.748110 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:31.748746 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:31.748658 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:41.747987 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:41.747946 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:04:51.748309 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:04:51.748269 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:05:01.748754 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:01.748711 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:05:11.748054 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:11.748018 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Jul 28 17:05:21.748582 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:21.748544 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:05:31.885284 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:31.885246 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:05:31.885858 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:31.885564 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" containerID="cri-o://f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f" gracePeriod=30 Jul 28 17:05:31.885858 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:31.885610 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kube-rbac-proxy" containerID="cri-o://ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7" gracePeriod=30 Jul 28 17:05:32.049188 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.049095 2575 generic.go:356] "Generic (PLEG): container finished" podID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerID="ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7" exitCode=2 Jul 28 17:05:32.049188 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.049175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerDied","Data":"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7"} Jul 28 17:05:32.936004 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.935965 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:32.939339 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.939316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:32.941104 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.941082 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\"" Jul 28 17:05:32.941261 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.941244 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\"" Jul 28 17:05:32.948900 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:32.948871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:33.035070 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.035026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6csf\" (UniqueName: \"kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.035070 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.035075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.035339 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.035135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.035339 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.035193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.135752 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.135717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6csf\" (UniqueName: \"kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.135752 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.135757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.136007 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.135891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.136007 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.135956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.136162 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:33.136143 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found Jul 28 17:05:33.136218 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.136196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.136255 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:33.136213 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls podName:a308d37d-48ba-43ab-a986-be090f66fc51 nodeName:}" failed. No retries permitted until 2026-07-28 17:05:33.636192381 +0000 UTC m=+5551.837039858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" (UID: "a308d37d-48ba-43ab-a986-be090f66fc51") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found Jul 28 17:05:33.136608 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.136592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.145055 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.145026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6csf\" (UniqueName: \"kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.640156 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.640113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.642531 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.642503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.851072 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.851030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:33.985812 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:33.985784 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:33.987884 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:05:33.987856 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda308d37d_48ba_43ab_a986_be090f66fc51.slice/crio-47fe37872b8e5f160bb431906fb16dcfafe8cd0f3507ee703f474810ae45b2f0 WatchSource:0}: Error finding container 47fe37872b8e5f160bb431906fb16dcfafe8cd0f3507ee703f474810ae45b2f0: Status 404 returned error can't find the container with id 47fe37872b8e5f160bb431906fb16dcfafe8cd0f3507ee703f474810ae45b2f0 Jul 28 17:05:34.057460 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:34.057432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerStarted","Data":"47fe37872b8e5f160bb431906fb16dcfafe8cd0f3507ee703f474810ae45b2f0"} Jul 28 17:05:35.062528 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:35.062492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerStarted","Data":"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761"} Jul 28 17:05:36.743650 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:36.743604 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused" Jul 28 17:05:37.029586 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.029556 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:05:37.071417 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.071363 2575 generic.go:356] "Generic (PLEG): container finished" podID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerID="f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f" exitCode=0 Jul 28 17:05:37.071593 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.071426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerDied","Data":"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f"} Jul 28 17:05:37.071593 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.071444 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" Jul 28 17:05:37.071593 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.071476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4" event={"ID":"e416d26b-b426-4308-b0f1-4fac932bd6cc","Type":"ContainerDied","Data":"7453b78eb768ba4dd8ccaf0b3c7a22e24209858146887c3e2a895d8ef2345fa0"} Jul 28 17:05:37.071593 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.071503 2575 scope.go:117] "RemoveContainer" containerID="ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7" Jul 28 17:05:37.078973 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.078951 2575 scope.go:117] "RemoveContainer" containerID="f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f" Jul 28 17:05:37.086222 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.086203 2575 scope.go:117] "RemoveContainer" containerID="e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315" Jul 28 17:05:37.093098 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093081 2575 scope.go:117] "RemoveContainer" containerID="ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7" Jul 28 17:05:37.093362 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:37.093343 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7\": container with ID starting with ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7 not found: ID does not exist" containerID="ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7" Jul 28 17:05:37.093437 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093372 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7"} err="failed to get container status \"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7\": rpc error: code = NotFound desc = could not find container \"ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7\": container with ID starting with ce8f60a114a5b6aaddefdd43d66cd7e05036066ad22c3497461ebaa8946d6bb7 not found: ID does not exist" Jul 28 17:05:37.093437 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093404 2575 scope.go:117] "RemoveContainer" containerID="f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f" Jul 28 17:05:37.093653 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:37.093637 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f\": container with ID starting with f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f not found: ID does not exist" containerID="f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f" Jul 28 17:05:37.093689 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093659 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f"} err="failed to get container status \"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f\": rpc error: code = NotFound desc = could not find container \"f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f\": container with ID starting with f798cf919d52f1826d19cb8b48a0594d7b43d48c498d3fa0df7a24cc97d5b67f not found: ID does not exist" Jul 28 17:05:37.093689 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093674 2575 scope.go:117] "RemoveContainer" containerID="e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315" Jul 28 17:05:37.093911 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:37.093894 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315\": container with ID starting with e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315 not found: ID does not exist" containerID="e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315" Jul 28 17:05:37.093965 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.093915 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315"} err="failed to get container status \"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315\": rpc error: code = NotFound desc = could not find container \"e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315\": container with ID starting with e68b4586a4777809b6ca2adcef7c4f75d32a877582acb589be0cc8682846a315 not found: ID does not exist" Jul 28 17:05:37.166872 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.166843 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls\") pod \"e416d26b-b426-4308-b0f1-4fac932bd6cc\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " Jul 28 17:05:37.166872 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.166878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert\") pod \"e416d26b-b426-4308-b0f1-4fac932bd6cc\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " Jul 28 17:05:37.167063 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.166897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config\") pod \"e416d26b-b426-4308-b0f1-4fac932bd6cc\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " Jul 28 17:05:37.167063 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.166941 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmz5p\" (UniqueName: \"kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p\") pod \"e416d26b-b426-4308-b0f1-4fac932bd6cc\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " Jul 28 17:05:37.167063 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.166992 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location\") pod \"e416d26b-b426-4308-b0f1-4fac932bd6cc\" (UID: \"e416d26b-b426-4308-b0f1-4fac932bd6cc\") " Jul 28 17:05:37.167255 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.167217 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e416d26b-b426-4308-b0f1-4fac932bd6cc" (UID: "e416d26b-b426-4308-b0f1-4fac932bd6cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:05:37.167382 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.167353 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e416d26b-b426-4308-b0f1-4fac932bd6cc" (UID: "e416d26b-b426-4308-b0f1-4fac932bd6cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:05:37.167382 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.167366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e416d26b-b426-4308-b0f1-4fac932bd6cc" (UID: "e416d26b-b426-4308-b0f1-4fac932bd6cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:05:37.168920 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.168899 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e416d26b-b426-4308-b0f1-4fac932bd6cc" (UID: "e416d26b-b426-4308-b0f1-4fac932bd6cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:05:37.169012 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.168992 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p" (OuterVolumeSpecName: "kube-api-access-xmz5p") pod "e416d26b-b426-4308-b0f1-4fac932bd6cc" (UID: "e416d26b-b426-4308-b0f1-4fac932bd6cc"). InnerVolumeSpecName "kube-api-access-xmz5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:05:37.267576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.267497 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e416d26b-b426-4308-b0f1-4fac932bd6cc-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:37.267576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.267528 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-cabundle-cert\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:37.267576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.267540 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:37.267576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.267548 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmz5p\" (UniqueName: \"kubernetes.io/projected/e416d26b-b426-4308-b0f1-4fac932bd6cc-kube-api-access-xmz5p\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:37.267576 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.267557 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e416d26b-b426-4308-b0f1-4fac932bd6cc-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:37.421235 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.421205 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:05:37.426364 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:37.426336 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-cvpq4"] Jul 28 17:05:38.076426 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:38.076334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/0.log" Jul 28 17:05:38.076426 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:38.076372 2575 generic.go:356] "Generic (PLEG): container finished" podID="a308d37d-48ba-43ab-a986-be090f66fc51" containerID="13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761" exitCode=1 Jul 28 17:05:38.076898 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:38.076449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerDied","Data":"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761"} Jul 28 17:05:38.372279 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:38.372197 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" path="/var/lib/kubelet/pods/e416d26b-b426-4308-b0f1-4fac932bd6cc/volumes" Jul 28 17:05:39.085306 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:39.085279 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/0.log" Jul 28 17:05:39.085672 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:39.085366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerStarted","Data":"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a"} Jul 28 17:05:42.917247 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:42.917196 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:42.917897 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:42.917476 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" containerID="cri-o://8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a" gracePeriod=30 Jul 28 17:05:44.459157 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.459133 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/1.log" Jul 28 17:05:44.459530 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.459516 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/0.log" Jul 28 17:05:44.461854 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.461839 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:44.526087 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config\") pod \"a308d37d-48ba-43ab-a986-be090f66fc51\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " Jul 28 17:05:44.526253 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526105 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location\") pod \"a308d37d-48ba-43ab-a986-be090f66fc51\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " Jul 28 17:05:44.526253 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526147 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6csf\" (UniqueName: \"kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf\") pod \"a308d37d-48ba-43ab-a986-be090f66fc51\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " Jul 28 17:05:44.526253 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526228 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") pod \"a308d37d-48ba-43ab-a986-be090f66fc51\" (UID: \"a308d37d-48ba-43ab-a986-be090f66fc51\") " Jul 28 17:05:44.526443 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526415 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a308d37d-48ba-43ab-a986-be090f66fc51" (UID: "a308d37d-48ba-43ab-a986-be090f66fc51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:05:44.526497 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.526467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a308d37d-48ba-43ab-a986-be090f66fc51" (UID: "a308d37d-48ba-43ab-a986-be090f66fc51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:05:44.528322 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.528298 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a308d37d-48ba-43ab-a986-be090f66fc51" (UID: "a308d37d-48ba-43ab-a986-be090f66fc51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:05:44.528431 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.528342 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf" (OuterVolumeSpecName: "kube-api-access-c6csf") pod "a308d37d-48ba-43ab-a986-be090f66fc51" (UID: "a308d37d-48ba-43ab-a986-be090f66fc51"). InnerVolumeSpecName "kube-api-access-c6csf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:05:44.627081 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.627042 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a308d37d-48ba-43ab-a986-be090f66fc51-proxy-tls\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:44.627081 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.627075 2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a308d37d-48ba-43ab-a986-be090f66fc51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:44.627081 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.627088 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a308d37d-48ba-43ab-a986-be090f66fc51-kserve-provision-location\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:44.627081 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:44.627097 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6csf\" (UniqueName: \"kubernetes.io/projected/a308d37d-48ba-43ab-a986-be090f66fc51-kube-api-access-c6csf\") on node \"ip-10-0-136-8.ec2.internal\" DevicePath \"\"" Jul 28 17:05:45.106538 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.106511 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/1.log" Jul 28 17:05:45.106894 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.106878 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49_a308d37d-48ba-43ab-a986-be090f66fc51/storage-initializer/0.log" Jul 28 17:05:45.106961 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.106919 2575 generic.go:356] "Generic (PLEG): container finished" podID="a308d37d-48ba-43ab-a986-be090f66fc51" containerID="8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a" exitCode=1 Jul 28 17:05:45.107009 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.106980 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" Jul 28 17:05:45.107009 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.106997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerDied","Data":"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a"} Jul 28 17:05:45.107084 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.107043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49" event={"ID":"a308d37d-48ba-43ab-a986-be090f66fc51","Type":"ContainerDied","Data":"47fe37872b8e5f160bb431906fb16dcfafe8cd0f3507ee703f474810ae45b2f0"} Jul 28 17:05:45.107084 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.107068 2575 scope.go:117] "RemoveContainer" containerID="8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a" Jul 28 17:05:45.115213 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.115197 2575 scope.go:117] "RemoveContainer" containerID="13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761" Jul 28 17:05:45.122371 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.122353 2575 scope.go:117] "RemoveContainer" containerID="8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a" Jul 28 17:05:45.122638 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:45.122619 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a\": container with ID starting with 8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a not found: ID does not exist" containerID="8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a" Jul 28 17:05:45.122696 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.122646 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a"} err="failed to get container status \"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a\": rpc error: code = NotFound desc = could not find container \"8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a\": container with ID starting with 8e4778f97e67f99991df34d4b90744b6d7c47a4eda16ab6d92366153add4cb8a not found: ID does not exist" Jul 28 17:05:45.122696 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.122662 2575 scope.go:117] "RemoveContainer" containerID="13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761" Jul 28 17:05:45.122860 ip-10-0-136-8 kubenswrapper[2575]: E0728 17:05:45.122845 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761\": container with ID starting with 13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761 not found: ID does not exist" containerID="13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761" Jul 28 17:05:45.122902 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.122861 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761"} err="failed to get container status \"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761\": rpc error: code = NotFound desc = could not find container \"13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761\": container with ID starting with 13bf54f80cd35b0507510f658d1590dad1023a6abedacd6ea5a74d7e555d1761 not found: ID does not exist" Jul 28 17:05:45.143185 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.143151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:45.149828 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:45.149801 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-pts49"] Jul 28 17:05:46.372197 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:05:46.372165 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" path="/var/lib/kubelet/pods/a308d37d-48ba-43ab-a986-be090f66fc51/volumes" Jul 28 17:06:20.709014 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.708975 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qccvn/must-gather-lfdzk"] Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709311 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kube-rbac-proxy" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709330 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kube-rbac-proxy" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709344 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709352 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709363 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709371 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709415 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709424 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709497 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709509 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kserve-container" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709518 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e416d26b-b426-4308-b0f1-4fac932bd6cc" containerName="kube-rbac-proxy" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709573 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.709637 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709583 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.710268 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.709650 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a308d37d-48ba-43ab-a986-be090f66fc51" containerName="storage-initializer" Jul 28 17:06:20.714120 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.714100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.716825 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.716801 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qccvn\"/\"default-dockercfg-lxx9f\"" Jul 28 17:06:20.716948 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.716895 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qccvn\"/\"openshift-service-ca.crt\"" Jul 28 17:06:20.716948 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.716919 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qccvn\"/\"kube-root-ca.crt\"" Jul 28 17:06:20.719424 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.719385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/must-gather-lfdzk"] Jul 28 17:06:20.797470 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.797423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-must-gather-output\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.797650 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.797492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrr4\" (UniqueName: \"kubernetes.io/projected/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-kube-api-access-mdrr4\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.898848 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.898815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-must-gather-output\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.899007 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.898867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrr4\" (UniqueName: \"kubernetes.io/projected/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-kube-api-access-mdrr4\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.899210 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.899187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-must-gather-output\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:20.907312 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:20.907282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrr4\" (UniqueName: \"kubernetes.io/projected/f33ad0dc-017b-47ef-9a36-4725c8fdd50a-kube-api-access-mdrr4\") pod \"must-gather-lfdzk\" (UID: \"f33ad0dc-017b-47ef-9a36-4725c8fdd50a\") " pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:21.024169 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:21.024115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/must-gather-lfdzk" Jul 28 17:06:21.152767 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:21.152735 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/must-gather-lfdzk"] Jul 28 17:06:21.155860 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:06:21.155821 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf33ad0dc_017b_47ef_9a36_4725c8fdd50a.slice/crio-05c4dd99a078800ff030a041fedda90b9b7a420ea36036b7d0b36564b3be41b8 WatchSource:0}: Error finding container 05c4dd99a078800ff030a041fedda90b9b7a420ea36036b7d0b36564b3be41b8: Status 404 returned error can't find the container with id 05c4dd99a078800ff030a041fedda90b9b7a420ea36036b7d0b36564b3be41b8 Jul 28 17:06:21.157670 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:21.157653 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:06:21.231782 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:21.231744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/must-gather-lfdzk" event={"ID":"f33ad0dc-017b-47ef-9a36-4725c8fdd50a","Type":"ContainerStarted","Data":"05c4dd99a078800ff030a041fedda90b9b7a420ea36036b7d0b36564b3be41b8"} Jul 28 17:06:22.238527 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:22.238423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/must-gather-lfdzk" event={"ID":"f33ad0dc-017b-47ef-9a36-4725c8fdd50a","Type":"ContainerStarted","Data":"8877c19fe881e8738e7e1fd6559f4efc1a75cd6149bbffe7c8306f9e925bfe7a"} Jul 28 17:06:22.238527 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:22.238474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/must-gather-lfdzk" event={"ID":"f33ad0dc-017b-47ef-9a36-4725c8fdd50a","Type":"ContainerStarted","Data":"5df36acec34b89db4566c1f2566b493cfebad55eeebb4646242844d7d8bc6204"} Jul 28 17:06:22.256505 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:22.256439 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qccvn/must-gather-lfdzk" podStartSLOduration=1.512976718 podStartE2EDuration="2.25641529s" totalImagesPullingTime="743.438572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:06:22.255749401 +0000 UTC m=+5600.456596893" watchObservedRunningTime="2026-07-28 17:06:22.25641529 +0000 UTC m=+5600.457262776" Jul 28 17:06:23.610335 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:23.610297 2575 ???:1] "http: TLS handshake error from 10.0.140.33:58710: EOF" Jul 28 17:06:23.624913 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:23.624851 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wgwzd_43a7b6a7-b8df-412b-82fa-ba25e8ea512f/global-pull-secret-syncer/0.log" Jul 28 17:06:23.763292 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:23.763254 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fr47v_99abd8e8-0dd4-4277-82cd-2f94b82b6f33/konnectivity-agent/0.log" Jul 28 17:06:23.855638 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:23.855611 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-8.ec2.internal_82fc34b6950a9b1738a19b765063a610/haproxy/0.log" Jul 28 17:06:28.376424 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:28.376366 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk8vw_2ef38ff7-f7b9-4935-a387-6ae9f0e25542/node-exporter/0.log" Jul 28 17:06:28.416046 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:28.416019 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk8vw_2ef38ff7-f7b9-4935-a387-6ae9f0e25542/kube-rbac-proxy/0.log" Jul 28 17:06:28.450587 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:28.450559 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk8vw_2ef38ff7-f7b9-4935-a387-6ae9f0e25542/init-textfile/0.log" Jul 28 17:06:31.462101 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:31.462074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66755c677b-tst75_dbabdc75-6227-492f-aede-af87d3ef8158/console/0.log" Jul 28 17:06:32.064884 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.064795 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546"] Jul 28 17:06:32.069605 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.069574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.086993 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.082774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546"] Jul 28 17:06:32.196308 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.196262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-podres\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.196654 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.196631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svgg\" (UniqueName: \"kubernetes.io/projected/0329ab22-a444-448c-9fb7-32faa6d09a05-kube-api-access-7svgg\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.197050 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.197030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-proc\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.197198 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.197177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-sys\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.197281 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.197226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-lib-modules\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298586 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-podres\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svgg\" (UniqueName: \"kubernetes.io/projected/0329ab22-a444-448c-9fb7-32faa6d09a05-kube-api-access-7svgg\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-proc\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-sys\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-lib-modules\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298702 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-proc\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-podres\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-sys\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.298780 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.298763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0329ab22-a444-448c-9fb7-32faa6d09a05-lib-modules\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.307352 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.307035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svgg\" (UniqueName: \"kubernetes.io/projected/0329ab22-a444-448c-9fb7-32faa6d09a05-kube-api-access-7svgg\") pod \"perf-node-gather-daemonset-s8546\" (UID: \"0329ab22-a444-448c-9fb7-32faa6d09a05\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.391811 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.391728 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:32.550068 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:32.549992 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546"] Jul 28 17:06:32.553487 ip-10-0-136-8 kubenswrapper[2575]: W0728 17:06:32.553449 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0329ab22_a444_448c_9fb7_32faa6d09a05.slice/crio-f9df9df3a6acfb4b3847a922bb447f0c41ede8985431852c06ae64163868f9ef WatchSource:0}: Error finding container f9df9df3a6acfb4b3847a922bb447f0c41ede8985431852c06ae64163868f9ef: Status 404 returned error can't find the container with id f9df9df3a6acfb4b3847a922bb447f0c41ede8985431852c06ae64163868f9ef Jul 28 17:06:33.161580 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.161552 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kzq4z_1ff3ce04-c9f0-4b53-86c6-4629f7a71927/dns/0.log" Jul 28 17:06:33.186930 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.186896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kzq4z_1ff3ce04-c9f0-4b53-86c6-4629f7a71927/kube-rbac-proxy/0.log" Jul 28 17:06:33.275692 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.275667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mb5q9_02fe1d62-7f94-4b61-ab13-e8bdeaea5870/dns-node-resolver/0.log" Jul 28 17:06:33.296328 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.296289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" event={"ID":"0329ab22-a444-448c-9fb7-32faa6d09a05","Type":"ContainerStarted","Data":"a103fd43000ac941741621cf6ec7140f899eb942b6d5b7837db43fc834d2bcd2"} Jul 28 17:06:33.296521 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.296356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" event={"ID":"0329ab22-a444-448c-9fb7-32faa6d09a05","Type":"ContainerStarted","Data":"f9df9df3a6acfb4b3847a922bb447f0c41ede8985431852c06ae64163868f9ef"} Jul 28 17:06:33.296521 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.296385 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:33.313232 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.313168 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" podStartSLOduration=1.313144858 podStartE2EDuration="1.313144858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:06:33.311045058 +0000 UTC m=+5611.511892562" watchObservedRunningTime="2026-07-28 17:06:33.313144858 +0000 UTC m=+5611.513992350" Jul 28 17:06:33.945500 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:33.945455 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85cdd7bbb8-5886g_41b39437-1f55-4ca0-8b3d-dafd0f6570bf/registry/0.log" Jul 28 17:06:34.039772 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:34.039743 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sj6zz_01e170ce-67ac-4dca-832a-ab31aa9991ce/node-ca/0.log" Jul 28 17:06:35.304444 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:35.304379 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5br4j_37f00501-a051-4585-9d34-0e25eac0aa3f/serve-healthcheck-canary/0.log" Jul 28 17:06:35.974556 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:35.974514 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zk6l7_f5c0f11b-9b4d-4024-ae55-e56eb381c5da/kube-rbac-proxy/0.log" Jul 28 17:06:36.003674 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:36.003648 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zk6l7_f5c0f11b-9b4d-4024-ae55-e56eb381c5da/exporter/0.log" Jul 28 17:06:36.037573 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:36.037541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zk6l7_f5c0f11b-9b4d-4024-ae55-e56eb381c5da/extractor/0.log" Jul 28 17:06:38.588367 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:38.588337 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-5q6mw_6fc9d1de-152f-4a06-9361-d1da024a57cd/manager/0.log" Jul 28 17:06:38.964892 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:38.964801 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-zvg49_37ffa276-f1bc-4f01-af7c-8a88a2c7f5db/manager/0.log" Jul 28 17:06:39.310296 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:39.310264 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-s8546" Jul 28 17:06:45.313608 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.313580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6vpqn_d891e983-ecf4-43b5-a414-942ab4c98826/kube-multus/0.log" Jul 28 17:06:45.561546 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.561522 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/kube-multus-additional-cni-plugins/0.log" Jul 28 17:06:45.587564 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.587482 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/egress-router-binary-copy/0.log" Jul 28 17:06:45.613171 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.613137 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/cni-plugins/0.log" Jul 28 17:06:45.637467 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.637439 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/bond-cni-plugin/0.log" Jul 28 17:06:45.661850 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.661826 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/routeoverride-cni/0.log" Jul 28 17:06:45.687690 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.687667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/whereabouts-cni-bincopy/0.log" Jul 28 17:06:45.717378 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:45.717348 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxmtr_5d0a302e-ec88-4382-9476-213b11b253f4/whereabouts-cni/0.log" Jul 28 17:06:46.021770 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:46.021742 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g965z_30fe3678-fed3-49a2-8b35-1eed25e9c327/network-metrics-daemon/0.log" Jul 28 17:06:46.049002 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:46.048974 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g965z_30fe3678-fed3-49a2-8b35-1eed25e9c327/kube-rbac-proxy/0.log" Jul 28 17:06:47.651126 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.651100 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/ovn-controller/0.log" Jul 28 17:06:47.675842 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.675813 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/ovn-acl-logging/0.log" Jul 28 17:06:47.708444 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.708412 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/kube-rbac-proxy-node/0.log" Jul 28 17:06:47.739039 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.739011 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 17:06:47.763927 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.763899 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/northd/0.log" Jul 28 17:06:47.799554 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.799525 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/nbdb/0.log" Jul 28 17:06:47.826246 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.826221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/sbdb/0.log" Jul 28 17:06:47.885444 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:47.885409 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgsxf_52a87494-68cd-4205-96cf-57bebf61dcdf/ovnkube-controller/0.log" Jul 28 17:06:48.991122 ip-10-0-136-8 kubenswrapper[2575]: I0728 17:06:48.991038 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7j2v2_505bfc5b-a2ad-4258-b7cd-d2c8ee8b4c27/network-check-target-container/0.log"