Sep 16 17:22:54.942384 ip-10-0-130-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 17:22:54.942396 ip-10-0-130-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 17:22:54.942403 ip-10-0-130-222 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 17:22:54.942566 ip-10-0-130-222 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 17:23:05.191507 ip-10-0-130-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 17:23:05.191522 ip-10-0-130-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2194985ca15c4c2fab4337e6fc606ab1 -- Sep 16 17:24:51.947236 ip-10-0-130-222 systemd[1]: Starting Kubernetes Kubelet... Sep 16 17:24:52.303314 ip-10-0-130-222 kubenswrapper[2536]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 17:24:52.303314 ip-10-0-130-222 kubenswrapper[2536]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 17:24:52.303314 ip-10-0-130-222 kubenswrapper[2536]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 17:24:52.303314 ip-10-0-130-222 kubenswrapper[2536]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 17:24:52.303314 ip-10-0-130-222 kubenswrapper[2536]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 17:24:52.304288 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.304198 2536 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 17:24:52.307810 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307793 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.307810 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307810 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307814 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307817 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307820 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307823 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307825 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307828 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307830 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307833 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307835 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307838 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307841 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307843 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307846 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307848 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307851 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307858 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307861 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307864 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307866 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.307874 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307868 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307871 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307873 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307876 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307878 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307881 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307885 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307888 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307892 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307894 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307897 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307902 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307905 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307908 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307921 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307924 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307927 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307929 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307932 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.308381 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307934 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307937 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307940 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307942 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307945 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307948 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307950 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307953 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307955 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307958 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307961 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307965 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307967 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307970 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307974 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307977 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307979 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307982 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307985 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307987 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.308822 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307989 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307992 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307994 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.307997 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308000 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308002 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308004 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308007 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308011 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308014 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308017 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308019 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308021 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308024 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308027 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308030 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308033 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308036 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308039 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.309352 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308042 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308046 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308049 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308052 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308055 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308057 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308061 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308063 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308067 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308070 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308073 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308076 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308078 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308081 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308083 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308086 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308089 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308091 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308094 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308097 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.309833 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308770 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308775 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308778 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308781 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308783 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308786 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308788 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308791 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308793 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308796 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308799 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308803 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308807 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308810 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308812 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308815 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308817 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308820 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308822 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308825 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.310337 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308828 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308831 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308834 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308837 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308839 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308842 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308844 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308847 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308850 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308852 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308855 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308857 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308860 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308862 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308864 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308867 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308869 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308872 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308874 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308877 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.310828 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308879 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308881 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308884 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308886 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308889 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308891 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308894 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308897 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308900 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308902 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308905 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308908 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308911 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308929 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308932 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308935 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308937 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308940 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308942 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.311321 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308945 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308947 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308950 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308952 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308955 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308958 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308961 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308963 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308966 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308968 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308971 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308973 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308976 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308978 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308980 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308983 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308986 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308990 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308993 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.311772 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308996 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.308999 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309002 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309005 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309008 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309011 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309014 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309016 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309019 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309021 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309024 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309026 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309030 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309033 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309035 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309038 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309042 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309045 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309048 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.312299 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309051 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309054 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309417 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309422 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309425 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309428 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309433 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309436 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309439 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309441 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309444 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309446 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309449 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309452 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309455 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309457 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309459 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309462 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309465 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309467 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.312748 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309470 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309472 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309475 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309478 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309480 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309483 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309485 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309488 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309490 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309493 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309495 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309498 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309500 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309503 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309505 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309508 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309510 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309512 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309516 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309519 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.313245 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309521 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309524 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309526 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309528 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309531 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309534 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309537 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309539 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309542 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309544 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309547 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309549 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309553 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309557 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309559 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309561 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309565 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309567 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309571 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.313749 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309574 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309577 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309579 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309582 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309585 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309587 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309591 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309594 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309597 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309599 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309602 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309606 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309608 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309610 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309613 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309615 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309618 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309620 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309622 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.314298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309625 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309629 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309631 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309634 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309636 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309639 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309641 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309643 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309646 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309648 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309651 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309653 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309655 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309658 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309660 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309663 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309665 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309667 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309670 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309672 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.314761 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309675 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309677 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.309679 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309758 2536 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309796 2536 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309803 2536 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309808 2536 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309812 2536 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309815 2536 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309819 2536 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309824 2536 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309828 2536 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309831 2536 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309835 2536 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309838 2536 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309842 2536 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309845 2536 flags.go:64] FLAG: --cgroup-root="" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309848 2536 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309851 2536 flags.go:64] FLAG: --client-ca-file="" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309854 2536 flags.go:64] FLAG: --cloud-provider="external" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309856 2536 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309860 2536 flags.go:64] FLAG: --cluster-domain="" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309863 2536 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 17:24:52.315251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309866 2536 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309869 2536 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309872 2536 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309876 2536 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309879 2536 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309882 2536 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309885 2536 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309888 2536 flags.go:64] FLAG: --contention-profiling="false" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309890 2536 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309894 2536 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309897 2536 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309899 2536 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309903 2536 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309907 2536 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309910 2536 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309928 2536 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309932 2536 flags.go:64] FLAG: --enable-server="true" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309935 2536 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309940 2536 flags.go:64] FLAG: --event-burst="100" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309943 2536 flags.go:64] FLAG: --event-qps="50" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309947 2536 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309950 2536 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309953 2536 flags.go:64] FLAG: --eviction-hard="" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309957 2536 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 17:24:52.315798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309960 2536 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309963 2536 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309965 2536 flags.go:64] FLAG: --eviction-soft="" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309968 2536 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309978 2536 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309982 2536 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309985 2536 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309988 2536 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309991 2536 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309994 2536 flags.go:64] FLAG: --feature-gates="" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.309998 2536 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310001 2536 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310004 2536 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310007 2536 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310010 2536 flags.go:64] FLAG: --healthz-port="10248" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310013 2536 flags.go:64] FLAG: --help="false" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310016 2536 flags.go:64] FLAG: --hostname-override="ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310019 2536 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310022 2536 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310025 2536 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310028 2536 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310033 2536 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310036 2536 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 17:24:52.316389 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310038 2536 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310041 2536 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310044 2536 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310046 2536 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310049 2536 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310053 2536 flags.go:64] FLAG: --kube-reserved="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310056 2536 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310059 2536 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310062 2536 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310065 2536 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310068 2536 flags.go:64] FLAG: --lock-file="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310071 2536 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310074 2536 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310077 2536 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310082 2536 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310085 2536 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310088 2536 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310091 2536 flags.go:64] FLAG: --logging-format="text" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310094 2536 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310097 2536 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310099 2536 flags.go:64] FLAG: --manifest-url="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310102 2536 flags.go:64] FLAG: --manifest-url-header="" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310106 2536 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310109 2536 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310113 2536 flags.go:64] FLAG: --max-pods="110" Sep 16 17:24:52.316995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310116 2536 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310119 2536 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310121 2536 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310124 2536 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310127 2536 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310132 2536 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310134 2536 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310141 2536 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310144 2536 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310147 2536 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310150 2536 flags.go:64] FLAG: --pod-cidr="" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310152 2536 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310158 2536 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310162 2536 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310165 2536 flags.go:64] FLAG: --pods-per-core="0" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310168 2536 flags.go:64] FLAG: --port="10250" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310171 2536 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310174 2536 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f4cfd983bdfb3b91" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310177 2536 flags.go:64] FLAG: --qos-reserved="" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310180 2536 flags.go:64] FLAG: --read-only-port="10255" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310183 2536 flags.go:64] FLAG: --register-node="true" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310186 2536 flags.go:64] FLAG: --register-with-taints="" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310189 2536 flags.go:64] FLAG: --registry-burst="10" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310192 2536 flags.go:64] FLAG: --registry-qps="5" Sep 16 17:24:52.317594 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310195 2536 flags.go:64] FLAG: --reserved-cpus="" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310197 2536 flags.go:64] FLAG: --reserved-memory="" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310201 2536 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310204 2536 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310207 2536 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310210 2536 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310213 2536 flags.go:64] FLAG: --runonce="false" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310216 2536 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310219 2536 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310221 2536 flags.go:64] FLAG: --seccomp-default="false" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310224 2536 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310227 2536 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310230 2536 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310234 2536 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310237 2536 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310240 2536 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310243 2536 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310245 2536 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310249 2536 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310252 2536 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310254 2536 flags.go:64] FLAG: --system-cgroups="" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310259 2536 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310264 2536 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310267 2536 flags.go:64] FLAG: --tls-cert-file="" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310270 2536 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 17:24:52.318192 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310274 2536 flags.go:64] FLAG: --tls-min-version="" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310277 2536 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310279 2536 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310282 2536 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310285 2536 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310288 2536 flags.go:64] FLAG: --v="2" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310291 2536 flags.go:64] FLAG: --version="false" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310295 2536 flags.go:64] FLAG: --vmodule="" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310299 2536 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.310302 2536 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310425 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310429 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310433 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310437 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310440 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310443 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310445 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310448 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310451 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310454 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310458 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.318787 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310460 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310463 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310465 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310468 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310472 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310475 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310478 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310481 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310484 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310487 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310490 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310492 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310495 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310498 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310501 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310503 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310506 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310508 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310511 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.319295 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310513 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310516 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310518 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310521 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310523 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310525 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310528 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310530 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310533 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310535 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310538 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310540 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310544 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310547 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310549 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310551 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310554 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310556 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310559 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310561 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.319745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310563 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310567 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310569 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310572 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310576 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310578 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310581 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310584 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310587 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310590 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310592 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310595 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310597 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310600 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310602 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310605 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310616 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310618 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310621 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310624 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.320281 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310626 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310629 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310633 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310636 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310641 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310644 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310647 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310649 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310652 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310654 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310657 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310660 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310662 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310665 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310667 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310670 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310673 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310675 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310678 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310680 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.320751 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310683 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310685 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310688 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310690 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310693 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310695 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310698 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310700 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.310703 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.311250 2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318103 2536 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318117 2536 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318131 2536 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318136 2536 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318207 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.321335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318212 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318216 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318219 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318223 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318227 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318230 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318233 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318236 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318239 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318241 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318244 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318247 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318250 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318252 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318254 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318257 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318260 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318268 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318271 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318273 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.321697 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318276 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318279 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318282 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318284 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318287 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318289 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318292 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318295 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318299 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318303 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318306 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318308 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318311 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318314 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318316 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318319 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318322 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318324 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318327 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318330 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.322183 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318332 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318334 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318337 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318341 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318344 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318347 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318349 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318355 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318358 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318361 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318370 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318372 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318375 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318377 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318380 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318383 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318385 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318388 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318390 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.322665 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318393 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318396 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318399 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318402 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318404 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318407 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318409 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318412 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318414 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318416 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318419 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318422 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318424 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318427 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318429 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318432 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318434 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318437 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318439 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318442 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.323132 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318444 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318447 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318449 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318452 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318459 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318462 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318465 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318467 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318469 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318472 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318474 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318477 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318479 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318482 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318485 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318487 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318490 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318492 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.323612 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318495 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318500 2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318627 2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318632 2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318635 2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318638 2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318640 2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318643 2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318646 2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318648 2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318651 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318653 2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318656 2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318659 2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318661 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 17:24:52.324063 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318663 2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318666 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318668 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318671 2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318680 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318683 2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318685 2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318688 2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318690 2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318693 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318695 2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318698 2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318701 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318704 2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318706 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318709 2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318711 2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318714 2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318716 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 17:24:52.324451 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318719 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318721 2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318724 2536 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318727 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318729 2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318731 2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318734 2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318736 2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318739 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318741 2536 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318743 2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318746 2536 feature_gate.go:328] unrecognized feature gate: Example Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318748 2536 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318752 2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318756 2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318759 2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318767 2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318770 2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318784 2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318787 2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 17:24:52.324898 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318789 2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318792 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318795 2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318798 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318801 2536 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318804 2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318806 2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318809 2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318812 2536 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318814 2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318817 2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318819 2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318822 2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318824 2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318826 2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318829 2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318832 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318834 2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318836 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318839 2536 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 17:24:52.325485 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318842 2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318844 2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318846 2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318849 2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318851 2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318854 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318856 2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318859 2536 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318861 2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318863 2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318866 2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318874 2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318876 2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318879 2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318882 2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318885 2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318889 2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318893 2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318896 2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 17:24:52.325978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318899 2536 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318902 2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318905 2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318907 2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318910 2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318923 2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318926 2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:52.318929 2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.318933 2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.319046 2536 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.320947 2536 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.321967 2536 server.go:1019] "Starting client certificate rotation" Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.322085 2536 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 17:24:52.326421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.322122 2536 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 17:24:52.326728 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.324785 2536 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 17:24:52.329976 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.329943 2536 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 17:24:52.341552 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.341531 2536 log.go:25] "Validated CRI v1 runtime API" Sep 16 17:24:52.344340 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.344323 2536 log.go:25] "Validated CRI v1 image API" Sep 16 17:24:52.345209 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.345193 2536 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 17:24:52.348423 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.348404 2536 fs.go:135] Filesystem UUIDs: map[5170453c-cd1e-4d59-a3c2-c7458901c506:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b329aeb3-9a7d-4b73-bee1-e78cb27e9654:/dev/nvme0n1p3] Sep 16 17:24:52.348475 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.348423 2536 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 17:24:52.353405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.353279 2536 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:52.35200097 +0000 UTC m=+0.295322578 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3192999 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c62d28f5e717bd2c274ea3583cf11 SystemUUID:ec2c62d2-8f5e-717b-d2c2-74ea3583cf11 BootID:2194985c-a15c-4c2f-ab43-37e6fc606ab1 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:01:c7:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:01:c7:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:e3:a8:7b:ec:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 17:24:52.353405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.353399 2536 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 17:24:52.353527 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.353486 2536 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 17:24:52.356794 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.356749 2536 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 17:24:52.356904 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.356882 2536 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 17:24:52.356962 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.356796 2536 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 17:24:52.358212 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.358200 2536 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 17:24:52.358256 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.358215 2536 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 17:24:52.358256 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.358223 2536 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 17:24:52.358880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.358865 2536 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 17:24:52.359608 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.359599 2536 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 17:24:52.363064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363052 2536 state_mem.go:36] "Initialized new in-memory state store" Sep 16 17:24:52.363186 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363177 2536 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 17:24:52.363221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363209 2536 kubelet.go:486] "Attempting to sync node with API server" Sep 16 17:24:52.363254 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363221 2536 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 17:24:52.363254 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363237 2536 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 17:24:52.363254 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363250 2536 kubelet.go:398] "Adding apiserver pod source" Sep 16 17:24:52.363364 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363259 2536 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 17:24:52.363708 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363696 2536 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 17:24:52.363761 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.363738 2536 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 17:24:52.366036 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.366010 2536 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 17:24:52.367947 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.367933 2536 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 17:24:52.367995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.367964 2536 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 17:24:52.369874 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.369842 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 17:24:52.370004 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.369884 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 17:24:52.370184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370173 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 17:24:52.370221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370186 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 17:24:52.370221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370204 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 17:24:52.370221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370210 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370234 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370240 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370257 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370264 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370291 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 17:24:52.370307 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.370297 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 17:24:52.371162 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.371016 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 17:24:52.371162 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.371035 2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 17:24:52.373816 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.373783 2536 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 17:24:52.374074 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.374050 2536 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 17:24:52.374315 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.373827 2536 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 17:24:52.375873 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.375858 2536 server.go:1268] "Started kubelet" Sep 16 17:24:52.375999 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.375956 2536 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 17:24:52.376050 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.376018 2536 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 17:24:52.376390 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.376351 2536 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 17:24:52.377096 ip-10-0-130-222 systemd[1]: Started Kubernetes Kubelet. Sep 16 17:24:52.377306 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.377103 2536 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 17:24:52.378150 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.378138 2536 server.go:310] "Adding debug handlers to kubelet server" Sep 16 17:24:52.379716 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.379696 2536 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 17:24:52.381266 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.381251 2536 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 17:24:52.381333 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.381302 2536 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 17:24:52.381333 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.381311 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.382033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382017 2536 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 17:24:52.382159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382146 2536 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 17:24:52.382217 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382163 2536 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 17:24:52.382297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382262 2536 reconstruct.go:97] "Volume reconstruction finished" Sep 16 17:24:52.382297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382271 2536 reconciler.go:29] "Reconciler: start to sync state" Sep 16 17:24:52.382683 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382613 2536 factory.go:55] Registering systemd factory Sep 16 17:24:52.382683 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382630 2536 factory.go:223] Registration of the systemd container factory successfully Sep 16 17:24:52.382926 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382897 2536 factory.go:153] Registering CRI-O factory Sep 16 17:24:52.382988 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.382935 2536 factory.go:223] Registration of the crio container factory successfully Sep 16 17:24:52.383107 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.383003 2536 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 17:24:52.383107 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.383023 2536 factory.go:103] Registering Raw factory Sep 16 17:24:52.383107 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.383047 2536 manager.go:1196] Started watching for new ooms in manager Sep 16 17:24:52.383275 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.380992 2536 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-222.ec2.internal.18d5dd1114626b58 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-222.ec2.internal,UID:ip-10-0-130-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-222.ec2.internal,},FirstTimestamp:2026-09-16 17:24:52.375817048 +0000 UTC m=+0.319138641,LastTimestamp:2026-09-16 17:24:52.375817048 +0000 UTC m=+0.319138641,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-222.ec2.internal,}" Sep 16 17:24:52.383473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.383461 2536 manager.go:349] Starting recovery of all containers Sep 16 17:24:52.384509 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.384472 2536 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 17:24:52.384509 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.384492 2536 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 17:24:52.394814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.394784 2536 manager.go:354] Recovery completed Sep 16 17:24:52.397444 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.397417 2536 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dvc2q" Sep 16 17:24:52.399999 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.399863 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.402027 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402004 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.402100 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402052 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.402100 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402068 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.402727 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402711 2536 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 17:24:52.402865 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402846 2536 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 17:24:52.402990 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402871 2536 state_mem.go:36] "Initialized new in-memory state store" Sep 16 17:24:52.402990 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.402737 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.403944 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.403881 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.403944 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.403930 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.404057 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.403947 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.404330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.404313 2536 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dvc2q" Sep 16 17:24:52.404665 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.404583 2536 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-222.ec2.internal.18d5dd1115f26092 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-222.ec2.internal,UID:ip-10-0-130-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-222.ec2.internal,},FirstTimestamp:2026-09-16 17:24:52.40202869 +0000 UTC m=+0.345350285,LastTimestamp:2026-09-16 17:24:52.40202869 +0000 UTC m=+0.345350285,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-222.ec2.internal,}" Sep 16 17:24:52.405140 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.405123 2536 policy_none.go:49] "None policy: Start" Sep 16 17:24:52.405206 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.405144 2536 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 17:24:52.405206 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.405159 2536 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 17:24:52.407008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.406995 2536 policy_none.go:47] "Start" Sep 16 17:24:52.430653 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.430636 2536 manager.go:337] "Starting Device Plugin manager" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.430676 2536 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.430690 2536 server.go:87] "Starting device plugin registration server" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.431074 2536 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.431089 2536 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.431218 2536 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.431288 2536 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.431294 2536 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.431813 2536 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 17:24:52.456606 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.431866 2536 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.470278 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.470245 2536 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 17:24:52.471498 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.471479 2536 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 17:24:52.471604 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.471534 2536 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 17:24:52.471604 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.471562 2536 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 17:24:52.471712 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.471661 2536 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 17:24:52.474005 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.473987 2536 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:52.532023 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.531993 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.533099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.533083 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.533180 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.533113 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.533180 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.533124 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.533180 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.533150 2536 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.543051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.543031 2536 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.543170 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.543059 2536 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-222.ec2.internal\": node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.572387 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.572314 2536 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal"] Sep 16 17:24:52.572387 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.572379 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.573896 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.573879 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.574457 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.574445 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.574498 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.574474 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.574498 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.574486 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.576881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.576867 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.576994 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.576980 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.577046 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577011 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.577743 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577727 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.577819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577745 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.577819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577760 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.577819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577769 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.577819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577773 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.577819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.577784 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.580048 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.580028 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.580127 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.580058 2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 17:24:52.580829 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.580814 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 17:24:52.580898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.580840 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 17:24:52.580898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.580853 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 17:24:52.583426 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.583406 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.583532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.583439 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.583532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.583478 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cab83a51988543fe63346f62bfb0589-config\") pod \"kube-apiserver-proxy-ip-10-0-130-222.ec2.internal\" (UID: \"3cab83a51988543fe63346f62bfb0589\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.594455 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.594434 2536 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-222.ec2.internal\" not found" node="ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.598235 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.598218 2536 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-222.ec2.internal\" not found" node="ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.674549 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.674518 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.683849 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683825 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.683931 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683855 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.683931 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683872 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cab83a51988543fe63346f62bfb0589-config\") pod \"kube-apiserver-proxy-ip-10-0-130-222.ec2.internal\" (UID: \"3cab83a51988543fe63346f62bfb0589\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.684002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683928 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.684002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683947 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cab83a51988543fe63346f62bfb0589-config\") pod \"kube-apiserver-proxy-ip-10-0-130-222.ec2.internal\" (UID: \"3cab83a51988543fe63346f62bfb0589\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.684002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.683960 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f07d75336aa3372e5da98f816547cbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal\" (UID: \"3f07d75336aa3372e5da98f816547cbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.774767 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.774737 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.875517 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.875439 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:52.899961 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.899931 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.901039 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:52.901017 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:52.975849 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:52.975811 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:53.076357 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.076336 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:53.178670 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.178603 2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-222.ec2.internal\" not found" Sep 16 17:24:53.277329 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.277303 2536 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:53.281653 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.281636 2536 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" Sep 16 17:24:53.300899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.300878 2536 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 17:24:53.302565 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.302543 2536 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" Sep 16 17:24:53.312739 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.312710 2536 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 17:24:53.321482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.321461 2536 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 17:24:53.321615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.321594 2536 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 17:24:53.321671 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.321626 2536 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 17:24:53.364258 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.364232 2536 apiserver.go:52] "Watching apiserver" Sep 16 17:24:53.381444 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.381409 2536 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 17:24:53.383133 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.383110 2536 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 17:24:53.383496 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.383473 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-vxj45","openshift-network-diagnostics/network-check-target-b9pjl","openshift-network-operator/iptables-alerter-xnm8r","openshift-ovn-kubernetes/ovnkube-node-5ss65","kube-system/global-pull-secret-syncer-t677q","kube-system/konnectivity-agent-p4kfb","kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn","openshift-cluster-node-tuning-operator/tuned-rg2wb","openshift-image-registry/node-ca-z6qms","openshift-multus/multus-8xqfp","openshift-multus/multus-additional-cni-plugins-fm9qz","openshift-dns/node-resolver-g5jxl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal"] Sep 16 17:24:53.386704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.386678 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.386805 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.386781 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:24:53.389346 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.389326 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:53.389493 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.389396 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:24:53.392611 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.392596 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.395144 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.395125 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.397578 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.397559 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.397672 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.397627 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:24:53.399870 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.399851 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.402367 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.402346 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljxbp\"" Sep 16 17:24:53.402448 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.402370 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.404813 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.404722 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.407271 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.407249 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.408194 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408159 2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:52 +0000 UTC" deadline="2028-06-14 05:14:07.11170328 +0000 UTC" Sep 16 17:24:53.408262 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408195 2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15275h49m13.703513115s" Sep 16 17:24:53.408508 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408485 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.408601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408542 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 17:24:53.408601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408558 2536 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 17:24:53.408717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408686 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 17:24:53.408888 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.408875 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 17:24:53.409041 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409026 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 17:24:53.410002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409799 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.410002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409856 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 17:24:53.410002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409873 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.410002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409940 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.410247 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.410018 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.410247 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409873 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.410247 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.409948 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.410359 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.410313 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.410996 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411007 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411050 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-85bfj\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411063 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411083 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f5ts9\"" Sep 16 17:24:53.411183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411139 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw48l\"" Sep 16 17:24:53.411533 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.411274 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k5jh9\"" Sep 16 17:24:53.412509 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.412492 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.414442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.414413 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.414592 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.414570 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 17:24:53.414741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.414722 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.416058 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.416040 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.416291 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.416271 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 17:24:53.424245 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424218 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.424548 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424499 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 17:24:53.424667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424242 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.424771 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424723 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-npd6k\"" Sep 16 17:24:53.424850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424821 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 17:24:53.424915 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424898 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.424996 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.424961 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98scm\"" Sep 16 17:24:53.425899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.425878 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x8cfd\"" Sep 16 17:24:53.426010 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.425942 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 17:24:53.436795 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.436774 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 17:24:53.437752 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.437732 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 17:24:53.438116 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.438099 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mqrlw\"" Sep 16 17:24:53.452383 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.452366 2536 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxncg" Sep 16 17:24:53.465440 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.465416 2536 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxncg" Sep 16 17:24:53.483650 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.483628 2536 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 17:24:53.487012 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.486988 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-242rq\" (UniqueName: \"kubernetes.io/projected/cc9a49cd-c9f4-4596-80e4-da2f05d76230-kube-api-access-242rq\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487096 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487023 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a95030-580d-4d9f-b746-3359da873540-agent-certs\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.487096 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487060 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.487096 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487077 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.487096 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487093 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.487240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487107 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-host\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.487240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487143 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-k8s-cni-cncf-io\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.487240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487170 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-cnibin\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.487240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487190 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-bin\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.487240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487225 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-kubelet-config\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487247 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-systemd\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487263 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-lib-modules\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487284 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvrg\" (UniqueName: \"kubernetes.io/projected/7e336a64-28bc-443a-9705-7a2b9da2ec59-kube-api-access-njvrg\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487305 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f33bb888-0d54-4843-99a6-f88c642ef5aa-host-slash\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487320 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-dbus\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487334 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-sys\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487356 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-host\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487380 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-socket-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.487419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487408 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-etc-selinux\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487439 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-sys-fs\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487463 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-cni-binary-copy\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487480 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvk7\" (UniqueName: \"kubernetes.io/projected/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-kube-api-access-mjvk7\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487496 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487511 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-netns\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487525 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-modprobe-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487546 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-var-lib-kubelet\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487575 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprvm\" (UniqueName: \"kubernetes.io/projected/981fabd0-06a6-4d18-8fae-181cfa9cca16-kube-api-access-lprvm\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487602 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-kubelet\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487620 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487650 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487671 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-cnibin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487690 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-socket-dir-parent\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487725 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-binary-copy\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.487760 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487750 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487766 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-kubelet\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487795 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-netd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487835 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-config\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487870 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-system-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487899 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-bin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487940 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs62\" (UniqueName: \"kubernetes.io/projected/146dffd8-7263-4546-89e9-8a9b7dcc47fb-kube-api-access-djs62\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487968 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-env-overrides\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.487991 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-tuned\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488026 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-serviceca\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488055 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-multus\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488080 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-multus-certs\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488106 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146dffd8-7263-4546-89e9-8a9b7dcc47fb-tmp-dir\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488130 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-slash\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488161 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488179 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-conf\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.488447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488194 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-system-cni-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488213 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488232 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-ovn\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488249 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-script-lib\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488269 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lttnv\" (UniqueName: \"kubernetes.io/projected/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-kube-api-access-lttnv\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488295 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488342 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a95030-580d-4d9f-b746-3359da873540-konnectivity-ca\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488377 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-conf-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488406 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7zq\" (UniqueName: \"kubernetes.io/projected/f33bb888-0d54-4843-99a6-f88c642ef5aa-kube-api-access-vn7zq\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488431 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-etc-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488456 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-node-log\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488480 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-run\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488506 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488529 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-netns\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488553 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8f9g\" (UniqueName: \"kubernetes.io/projected/586a3038-36d1-4c2f-8739-97338f3663ba-kube-api-access-l8f9g\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488577 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-var-lib-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488608 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/146dffd8-7263-4546-89e9-8a9b7dcc47fb-hosts-file\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488635 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-systemd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488656 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-systemd-units\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488679 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-log-socket\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488709 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysconfig\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488761 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-device-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488786 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-kube-api-access-sp9st\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488819 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-etc-kubernetes\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488878 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f33bb888-0d54-4843-99a6-f88c642ef5aa-iptables-alerter-script\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488910 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488955 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.488980 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-registration-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.489003 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-hostroot\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.489044 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-tmp\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.489062 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-multus-daemon-config\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.489638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.489078 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-kubernetes\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.542946 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.542913 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cab83a51988543fe63346f62bfb0589.slice/crio-19fc177d7e68db5ee46bc78ffdbdd950909f613c78011ca83747575d70aebb93 WatchSource:0}: Error finding container 19fc177d7e68db5ee46bc78ffdbdd950909f613c78011ca83747575d70aebb93: Status 404 returned error can't find the container with id 19fc177d7e68db5ee46bc78ffdbdd950909f613c78011ca83747575d70aebb93 Sep 16 17:24:53.543195 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.543178 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07d75336aa3372e5da98f816547cbd.slice/crio-f7359a42c47ef5f59161628d598eeaec770dd5de5f2b35fb2c99e80ef9a97f35 WatchSource:0}: Error finding container f7359a42c47ef5f59161628d598eeaec770dd5de5f2b35fb2c99e80ef9a97f35: Status 404 returned error can't find the container with id f7359a42c47ef5f59161628d598eeaec770dd5de5f2b35fb2c99e80ef9a97f35 Sep 16 17:24:53.550290 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.550264 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 17:24:53.589952 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.589910 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-kubernetes\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.589952 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.589955 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-242rq\" (UniqueName: \"kubernetes.io/projected/cc9a49cd-c9f4-4596-80e4-da2f05d76230-kube-api-access-242rq\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.589971 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a95030-580d-4d9f-b746-3359da873540-agent-certs\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.589987 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590087 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-kubernetes\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590123 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590158 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590182 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.590205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590183 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-host\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590211 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590219 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-host\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590234 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-k8s-cni-cncf-io\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590255 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-cnibin\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590251 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590271 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-bin\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590280 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-cnibin\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590256 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-k8s-cni-cncf-io\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590303 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-kubelet-config\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590307 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-bin\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590327 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-systemd\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590352 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-lib-modules\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590368 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-kubelet-config\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590378 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvrg\" (UniqueName: \"kubernetes.io/projected/7e336a64-28bc-443a-9705-7a2b9da2ec59-kube-api-access-njvrg\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590411 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f33bb888-0d54-4843-99a6-f88c642ef5aa-host-slash\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590411 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-systemd\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.590497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590447 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-dbus\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590452 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f33bb888-0d54-4843-99a6-f88c642ef5aa-host-slash\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590482 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-sys\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590500 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a839c16c-c00b-428c-8c7e-1b25b469967f-dbus\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590533 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-host\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590537 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-lib-modules\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590565 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-socket-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590584 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-host\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590593 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-etc-selinux\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590611 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-sys\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590619 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-sys-fs\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590633 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-etc-selinux\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590646 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-cni-binary-copy\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590664 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-sys-fs\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590686 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-socket-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590734 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvk7\" (UniqueName: \"kubernetes.io/projected/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-kube-api-access-mjvk7\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590753 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.591274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590770 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-netns\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590786 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-modprobe-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590811 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-var-lib-kubelet\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590813 2536 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590835 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprvm\" (UniqueName: \"kubernetes.io/projected/981fabd0-06a6-4d18-8fae-181cfa9cca16-kube-api-access-lprvm\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590859 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-kubelet\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590859 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-netns\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590917 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-kubelet\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.590928 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-var-lib-kubelet\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591055 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591090 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591121 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-modprobe-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591124 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-cnibin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591156 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-socket-dir-parent\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591171 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-cnibin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591183 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-binary-copy\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591209 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591227 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-socket-dir-parent\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591236 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-kubelet\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591277 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-kubelet\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.591318 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.591410 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.09136892 +0000 UTC m=+2.034690502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591423 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-netd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591444 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-config\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591469 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-system-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591490 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-bin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591504 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-cni-netd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591514 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs62\" (UniqueName: \"kubernetes.io/projected/146dffd8-7263-4546-89e9-8a9b7dcc47fb-kube-api-access-djs62\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591570 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-env-overrides\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591599 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-tuned\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591646 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-serviceca\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591672 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-multus\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591698 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-multus-certs\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591701 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591739 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146dffd8-7263-4546-89e9-8a9b7dcc47fb-tmp-dir\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.592975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591765 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-slash\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591791 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591836 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-conf\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591860 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-system-cni-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591899 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591916 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-cni-binary-copy\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591951 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-ovn\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591977 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-script-lib\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591991 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-slash\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592003 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lttnv\" (UniqueName: \"kubernetes.io/projected/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-kube-api-access-lttnv\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592030 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-multus-certs\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592054 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592062 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-env-overrides\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592078 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a95030-580d-4d9f-b746-3359da873540-konnectivity-ca\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592117 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-conf-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592148 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7zq\" (UniqueName: \"kubernetes.io/projected/f33bb888-0d54-4843-99a6-f88c642ef5aa-kube-api-access-vn7zq\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592176 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-etc-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.593799 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592206 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-node-log\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592244 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-config\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592249 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-run\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592274 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592296 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146dffd8-7263-4546-89e9-8a9b7dcc47fb-tmp-dir\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592308 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-system-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591738 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-binary-copy\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592344 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-conf-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592299 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-netns\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592381 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8f9g\" (UniqueName: \"kubernetes.io/projected/586a3038-36d1-4c2f-8739-97338f3663ba-kube-api-access-l8f9g\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592390 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-run-netns\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.591809 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-multus\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592411 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-var-lib-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592427 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-host-var-lib-cni-bin\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592437 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/146dffd8-7263-4546-89e9-8a9b7dcc47fb-hosts-file\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592479 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592491 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/146dffd8-7263-4546-89e9-8a9b7dcc47fb-hosts-file\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592515 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-conf\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.594378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592522 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-systemd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592561 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-systemd-units\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592572 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-node-log\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592566 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e336a64-28bc-443a-9705-7a2b9da2ec59-system-cni-dir\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592594 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-etc-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592610 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-log-socket\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592630 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-run\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592642 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-var-lib-openvswitch\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592641 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysconfig\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592679 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-systemd\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592679 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-device-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592690 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-multus-cni-dir\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592715 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-kube-api-access-sp9st\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592734 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysconfig\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592743 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-etc-kubernetes\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592744 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-serviceca\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592770 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f33bb888-0d54-4843-99a6-f88c642ef5aa-iptables-alerter-script\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592798 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592806 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-systemd-units\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592825 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592843 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-log-socket\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592851 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-registration-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592878 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-hostroot\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592886 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-etc-kubernetes\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592717 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-device-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592898 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a95030-580d-4d9f-b746-3359da873540-konnectivity-ca\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592904 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-tmp\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592936 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-run-ovn\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.592944 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-multus-daemon-config\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593011 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/586a3038-36d1-4c2f-8739-97338f3663ba-hostroot\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.593040 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.593104 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.093087739 +0000 UTC m=+2.036409324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593156 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593192 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e336a64-28bc-443a-9705-7a2b9da2ec59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593347 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-sysctl-d\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.595924 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593507 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovnkube-script-lib\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593580 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/981fabd0-06a6-4d18-8fae-181cfa9cca16-registration-dir\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593699 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f33bb888-0d54-4843-99a6-f88c642ef5aa-iptables-alerter-script\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.593777 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/586a3038-36d1-4c2f-8739-97338f3663ba-multus-daemon-config\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.595127 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-etc-tuned\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.595168 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.595369 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a95030-580d-4d9f-b746-3359da873540-agent-certs\") pod \"konnectivity-agent-p4kfb\" (UID: \"e3a95030-580d-4d9f-b746-3359da873540\") " pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.596485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.596202 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9a49cd-c9f4-4596-80e4-da2f05d76230-tmp\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.603147 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.603124 2536 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:53.603412 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.603392 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:24:53.603501 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.603417 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:24:53.603501 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.603434 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:53.603612 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:53.603511 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.103493122 +0000 UTC m=+2.046814705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:53.604718 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.604690 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprvm\" (UniqueName: \"kubernetes.io/projected/981fabd0-06a6-4d18-8fae-181cfa9cca16-kube-api-access-lprvm\") pod \"aws-ebs-csi-driver-node-sstxn\" (UID: \"981fabd0-06a6-4d18-8fae-181cfa9cca16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.604847 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.604751 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvk7\" (UniqueName: \"kubernetes.io/projected/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-kube-api-access-mjvk7\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:53.604948 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.604922 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-242rq\" (UniqueName: \"kubernetes.io/projected/cc9a49cd-c9f4-4596-80e4-da2f05d76230-kube-api-access-242rq\") pod \"tuned-rg2wb\" (UID: \"cc9a49cd-c9f4-4596-80e4-da2f05d76230\") " pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.605580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.605556 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvrg\" (UniqueName: \"kubernetes.io/projected/7e336a64-28bc-443a-9705-7a2b9da2ec59-kube-api-access-njvrg\") pod \"multus-additional-cni-plugins-fm9qz\" (UID: \"7e336a64-28bc-443a-9705-7a2b9da2ec59\") " pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.607090 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.607063 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8f9g\" (UniqueName: \"kubernetes.io/projected/586a3038-36d1-4c2f-8739-97338f3663ba-kube-api-access-l8f9g\") pod \"multus-8xqfp\" (UID: \"586a3038-36d1-4c2f-8739-97338f3663ba\") " pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.607185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.607113 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7zq\" (UniqueName: \"kubernetes.io/projected/f33bb888-0d54-4843-99a6-f88c642ef5aa-kube-api-access-vn7zq\") pod \"iptables-alerter-xnm8r\" (UID: \"f33bb888-0d54-4843-99a6-f88c642ef5aa\") " pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.607385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.607365 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs62\" (UniqueName: \"kubernetes.io/projected/146dffd8-7263-4546-89e9-8a9b7dcc47fb-kube-api-access-djs62\") pod \"node-resolver-g5jxl\" (UID: \"146dffd8-7263-4546-89e9-8a9b7dcc47fb\") " pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.607580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.607560 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9st\" (UniqueName: \"kubernetes.io/projected/df81b8c0-eb31-498b-af78-a67a0cf5b8b0-kube-api-access-sp9st\") pod \"node-ca-z6qms\" (UID: \"df81b8c0-eb31-498b-af78-a67a0cf5b8b0\") " pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.607690 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.607674 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lttnv\" (UniqueName: \"kubernetes.io/projected/2b5cbe64-815c-4b5f-91e5-2e90eb6939c0-kube-api-access-lttnv\") pod \"ovnkube-node-5ss65\" (UID: \"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.732234 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.732193 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xnm8r" Sep 16 17:24:53.740267 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.740248 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:24:53.743466 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.743436 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33bb888_0d54_4843_99a6_f88c642ef5aa.slice/crio-f5696a18a390a35a75b8f6b8d24397dddd7fe5cd0bc8cb22aa63dcce1de98e03 WatchSource:0}: Error finding container f5696a18a390a35a75b8f6b8d24397dddd7fe5cd0bc8cb22aa63dcce1de98e03: Status 404 returned error can't find the container with id f5696a18a390a35a75b8f6b8d24397dddd7fe5cd0bc8cb22aa63dcce1de98e03 Sep 16 17:24:53.751298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.751275 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5cbe64_815c_4b5f_91e5_2e90eb6939c0.slice/crio-b82615efda58657fb0c7fcde0e28e544fca873f75a8d6c7b4b2f77f435197201 WatchSource:0}: Error finding container b82615efda58657fb0c7fcde0e28e544fca873f75a8d6c7b4b2f77f435197201: Status 404 returned error can't find the container with id b82615efda58657fb0c7fcde0e28e544fca873f75a8d6c7b4b2f77f435197201 Sep 16 17:24:53.766253 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.766227 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:24:53.775628 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.775595 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a95030_580d_4d9f_b746_3359da873540.slice/crio-dd22941151f9bc79da6a68b504a37154b13b54a749687f063aecb059a4668ef2 WatchSource:0}: Error finding container dd22941151f9bc79da6a68b504a37154b13b54a749687f063aecb059a4668ef2: Status 404 returned error can't find the container with id dd22941151f9bc79da6a68b504a37154b13b54a749687f063aecb059a4668ef2 Sep 16 17:24:53.781869 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.781850 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" Sep 16 17:24:53.790229 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.790208 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981fabd0_06a6_4d18_8fae_181cfa9cca16.slice/crio-288820513b3df83679b1b2a1612f5e90857c0a3126957c59b7a4e10112a74ace WatchSource:0}: Error finding container 288820513b3df83679b1b2a1612f5e90857c0a3126957c59b7a4e10112a74ace: Status 404 returned error can't find the container with id 288820513b3df83679b1b2a1612f5e90857c0a3126957c59b7a4e10112a74ace Sep 16 17:24:53.795480 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.795459 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" Sep 16 17:24:53.803486 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.803464 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9a49cd_c9f4_4596_80e4_da2f05d76230.slice/crio-f21d0b6ba96f30b5e32d787b708e4559b0c37ba781bc1043f1d10f02cc5e0afb WatchSource:0}: Error finding container f21d0b6ba96f30b5e32d787b708e4559b0c37ba781bc1043f1d10f02cc5e0afb: Status 404 returned error can't find the container with id f21d0b6ba96f30b5e32d787b708e4559b0c37ba781bc1043f1d10f02cc5e0afb Sep 16 17:24:53.810570 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.810550 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6qms" Sep 16 17:24:53.815613 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.815597 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xqfp" Sep 16 17:24:53.819408 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.819249 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf81b8c0_eb31_498b_af78_a67a0cf5b8b0.slice/crio-7eed59dda32e5a3c144029d5847206b3e84b45fd48d3f96c2caa61e576adac54 WatchSource:0}: Error finding container 7eed59dda32e5a3c144029d5847206b3e84b45fd48d3f96c2caa61e576adac54: Status 404 returned error can't find the container with id 7eed59dda32e5a3c144029d5847206b3e84b45fd48d3f96c2caa61e576adac54 Sep 16 17:24:53.824742 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.824723 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586a3038_36d1_4c2f_8739_97338f3663ba.slice/crio-4777774aa24022b4776c14f5a908ced8d5876cfa5440ebad7ca4f1b03cf7994a WatchSource:0}: Error finding container 4777774aa24022b4776c14f5a908ced8d5876cfa5440ebad7ca4f1b03cf7994a: Status 404 returned error can't find the container with id 4777774aa24022b4776c14f5a908ced8d5876cfa5440ebad7ca4f1b03cf7994a Sep 16 17:24:53.841034 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.841007 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" Sep 16 17:24:53.851096 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.851072 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e336a64_28bc_443a_9705_7a2b9da2ec59.slice/crio-f496b782410e07c4442bdeef757c91063f56a2ce836c58717dc81e96c7e3a3c6 WatchSource:0}: Error finding container f496b782410e07c4442bdeef757c91063f56a2ce836c58717dc81e96c7e3a3c6: Status 404 returned error can't find the container with id f496b782410e07c4442bdeef757c91063f56a2ce836c58717dc81e96c7e3a3c6 Sep 16 17:24:53.851745 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.851726 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5jxl" Sep 16 17:24:53.861273 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:24:53.861253 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146dffd8_7263_4546_89e9_8a9b7dcc47fb.slice/crio-62987e05b919950b8bfde71d46947d1a660f7a46477ee3d1491746fb1edae900 WatchSource:0}: Error finding container 62987e05b919950b8bfde71d46947d1a660f7a46477ee3d1491746fb1edae900: Status 404 returned error can't find the container with id 62987e05b919950b8bfde71d46947d1a660f7a46477ee3d1491746fb1edae900 Sep 16 17:24:53.964346 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:53.964311 2536 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:54.096808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.096714 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:54.096808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.096806 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:54.097093 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.096920 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:54.097093 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.096920 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:54.097093 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.097008 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:24:55.096970242 +0000 UTC m=+3.040291830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:54.097093 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.097028 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:55.097018552 +0000 UTC m=+3.040340136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:54.198020 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.197983 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:54.198258 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.198188 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:24:54.198258 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.198212 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:24:54.198258 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.198226 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:54.198450 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:54.198285 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:55.19826706 +0000 UTC m=+3.141588645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:54.303341 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.303310 2536 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:54.473139 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.473013 2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:54 +0000 UTC" deadline="2028-03-02 15:57:44.021968659 +0000 UTC" Sep 16 17:24:54.473139 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.473076 2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12790h32m49.548897238s" Sep 16 17:24:54.504303 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.504154 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" event={"ID":"3f07d75336aa3372e5da98f816547cbd","Type":"ContainerStarted","Data":"f7359a42c47ef5f59161628d598eeaec770dd5de5f2b35fb2c99e80ef9a97f35"} Sep 16 17:24:54.519865 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.519826 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerStarted","Data":"f496b782410e07c4442bdeef757c91063f56a2ce836c58717dc81e96c7e3a3c6"} Sep 16 17:24:54.532689 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.532647 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6qms" event={"ID":"df81b8c0-eb31-498b-af78-a67a0cf5b8b0","Type":"ContainerStarted","Data":"7eed59dda32e5a3c144029d5847206b3e84b45fd48d3f96c2caa61e576adac54"} Sep 16 17:24:54.549485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.549444 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" event={"ID":"cc9a49cd-c9f4-4596-80e4-da2f05d76230","Type":"ContainerStarted","Data":"f21d0b6ba96f30b5e32d787b708e4559b0c37ba781bc1043f1d10f02cc5e0afb"} Sep 16 17:24:54.578413 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.578369 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" event={"ID":"981fabd0-06a6-4d18-8fae-181cfa9cca16","Type":"ContainerStarted","Data":"288820513b3df83679b1b2a1612f5e90857c0a3126957c59b7a4e10112a74ace"} Sep 16 17:24:54.582004 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.581955 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p4kfb" event={"ID":"e3a95030-580d-4d9f-b746-3359da873540","Type":"ContainerStarted","Data":"dd22941151f9bc79da6a68b504a37154b13b54a749687f063aecb059a4668ef2"} Sep 16 17:24:54.594527 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.594485 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xnm8r" event={"ID":"f33bb888-0d54-4843-99a6-f88c642ef5aa","Type":"ContainerStarted","Data":"f5696a18a390a35a75b8f6b8d24397dddd7fe5cd0bc8cb22aa63dcce1de98e03"} Sep 16 17:24:54.617629 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.617540 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" event={"ID":"3cab83a51988543fe63346f62bfb0589","Type":"ContainerStarted","Data":"19fc177d7e68db5ee46bc78ffdbdd950909f613c78011ca83747575d70aebb93"} Sep 16 17:24:54.625227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.625191 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5jxl" event={"ID":"146dffd8-7263-4546-89e9-8a9b7dcc47fb","Type":"ContainerStarted","Data":"62987e05b919950b8bfde71d46947d1a660f7a46477ee3d1491746fb1edae900"} Sep 16 17:24:54.646240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.646096 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xqfp" event={"ID":"586a3038-36d1-4c2f-8739-97338f3663ba","Type":"ContainerStarted","Data":"4777774aa24022b4776c14f5a908ced8d5876cfa5440ebad7ca4f1b03cf7994a"} Sep 16 17:24:54.664086 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.664042 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"b82615efda58657fb0c7fcde0e28e544fca873f75a8d6c7b4b2f77f435197201"} Sep 16 17:24:54.997194 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:54.997161 2536 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 17:24:55.111339 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.111298 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:55.111587 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.111364 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:55.111587 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.111497 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:55.111587 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.111505 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:55.111587 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.111563 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.111544592 +0000 UTC m=+5.054866189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:55.111587 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.111583 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.111573189 +0000 UTC m=+5.054894781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:55.212273 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.212231 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:55.212460 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.212443 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:24:55.212524 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.212469 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:24:55.212524 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.212484 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:55.212596 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.212546 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.212528187 +0000 UTC m=+5.155849775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:55.472275 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.472243 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:55.472472 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.472428 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:24:55.472950 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.472922 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:55.473145 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.473047 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:24:55.473219 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.473155 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:55.473593 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:55.473229 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:24:55.473593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.473334 2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:54 +0000 UTC" deadline="2028-03-04 19:26:35.577040321 +0000 UTC" Sep 16 17:24:55.473593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:55.473357 2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12842h1m40.103687451s" Sep 16 17:24:57.132797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.132579 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:57.132797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.132643 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:57.132797 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.132765 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:57.133417 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.132848 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:25:01.132827058 +0000 UTC m=+9.076148660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:24:57.133417 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.132776 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:57.133417 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.132979 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:01.132953028 +0000 UTC m=+9.076274626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:24:57.233838 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.233717 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:57.234056 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.233925 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:24:57.234056 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.233947 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:24:57.234056 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.233978 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:57.234242 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.234094 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:01.234069789 +0000 UTC m=+9.177391398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:24:57.472805 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.472766 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:57.473019 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.472780 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:57.473019 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.472907 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:24:57.473019 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:57.472780 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:57.473245 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.473058 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:24:57.473245 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:57.473077 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:24:59.472651 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:59.472614 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:24:59.473170 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:59.472831 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:24:59.473170 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:59.473006 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:24:59.473170 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:59.473101 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:24:59.473170 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:24:59.473160 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:24:59.473417 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:24:59.473254 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:01.170620 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.170579 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:01.170620 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.170627 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:01.171099 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.170732 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:01.171099 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.170794 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:09.170776547 +0000 UTC m=+17.114098143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:01.171099 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.171083 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:01.171252 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.171162 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:25:09.171138244 +0000 UTC m=+17.114459838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:01.271584 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.271545 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:01.272330 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.271752 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:25:01.272330 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.271778 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:25:01.272330 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.271791 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:01.272330 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.271847 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:09.271830186 +0000 UTC m=+17.215151778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:01.472756 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.472722 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:01.472982 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.472728 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:01.472982 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.472858 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:01.472982 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:01.472744 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:01.472982 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.472944 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:01.473212 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:01.473034 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:03.472259 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:03.472226 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:03.472826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:03.472348 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:03.472826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:03.472356 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:03.472826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:03.472385 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:03.472826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:03.472478 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:03.472826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:03.472574 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:05.471940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:05.471895 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:05.471940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:05.471934 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:05.472365 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:05.471908 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:05.472365 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:05.472015 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:05.472365 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:05.472119 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:05.472365 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:05.472209 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:07.472355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:07.472320 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:07.472802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:07.472367 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:07.472802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:07.472444 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:07.472802 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:07.472452 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:07.472802 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:07.472535 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:07.472802 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:07.472650 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:09.228568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.228536 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:09.228568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.228574 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:09.229127 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.228677 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:09.229127 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.228723 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:25.228710864 +0000 UTC m=+33.172032457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:09.229127 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.228678 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:09.229127 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.228806 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:25:25.22879201 +0000 UTC m=+33.172113604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:09.329875 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.329842 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:09.330050 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.330018 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:25:09.330050 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.330040 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:25:09.330148 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.330053 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:09.330148 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.330118 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:25.330102948 +0000 UTC m=+33.273424529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:09.472327 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.472289 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:09.472515 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.472339 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:09.472515 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.472416 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:09.472515 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:09.472307 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:09.472667 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.472537 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:09.472667 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:09.472647 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:11.472181 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:11.472141 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:11.472727 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:11.472187 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:11.472727 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:11.472190 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:11.472727 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:11.472267 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:11.472727 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:11.472376 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:11.472727 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:11.472459 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:12.721886 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:12.721640 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" event={"ID":"cc9a49cd-c9f4-4596-80e4-da2f05d76230","Type":"ContainerStarted","Data":"eca652a84ecf32ce22d78b5764df5f23dea864a1d93cd40b04383f866adf947a"} Sep 16 17:25:12.723296 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:12.723264 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" event={"ID":"3cab83a51988543fe63346f62bfb0589","Type":"ContainerStarted","Data":"8eb9e84b61ce87ab8f5039c3292ad1213f0ebafefba8eac91ad5c2d591318bec"} Sep 16 17:25:12.724865 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:12.724840 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"b6041b3841c8017b5f2d93e618dadcfa257a7a9f1a2208e9143bb6dadb4f21de"} Sep 16 17:25:12.800804 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:12.800746 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-222.ec2.internal" podStartSLOduration=19.800728985 podStartE2EDuration="19.800728985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:12.800730485 +0000 UTC m=+20.744052088" watchObservedRunningTime="2026-09-16 17:25:12.800728985 +0000 UTC m=+20.744050587" Sep 16 17:25:12.800910 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:12.800878 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rg2wb" podStartSLOduration=1.184431888 podStartE2EDuration="19.800872055s" totalImagesPullingTime="18.616440167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:12.761710697 +0000 UTC m=+20.705032298" watchObservedRunningTime="2026-09-16 17:25:12.800872055 +0000 UTC m=+20.744193659" Sep 16 17:25:13.472715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.472529 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:13.472843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.472574 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:13.472843 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:13.472793 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:13.472944 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.472578 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:13.472944 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:13.472868 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:13.473012 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:13.472975 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:13.728813 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.728781 2536 generic.go:356] "Generic (PLEG): container finished" podID="3f07d75336aa3372e5da98f816547cbd" containerID="789c47ca3ff6fd5f1ba7a846b97f1bfb032e639a8623c3ec11f341c308802e90" exitCode=0 Sep 16 17:25:13.729445 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.728871 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" event={"ID":"3f07d75336aa3372e5da98f816547cbd","Type":"ContainerDied","Data":"789c47ca3ff6fd5f1ba7a846b97f1bfb032e639a8623c3ec11f341c308802e90"} Sep 16 17:25:13.730414 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.730388 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="260d3affe95d2ed229d6189842c3d83e247556e56aa74e1ecad0865c522046ec" exitCode=0 Sep 16 17:25:13.730519 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.730453 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"260d3affe95d2ed229d6189842c3d83e247556e56aa74e1ecad0865c522046ec"} Sep 16 17:25:13.731975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.731954 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6qms" event={"ID":"df81b8c0-eb31-498b-af78-a67a0cf5b8b0","Type":"ContainerStarted","Data":"4fb2e12b608fa0c7cd194721bde838e18d2175e892a2a6f200a0b07fc924f892"} Sep 16 17:25:13.733493 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.733463 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" event={"ID":"981fabd0-06a6-4d18-8fae-181cfa9cca16","Type":"ContainerStarted","Data":"c73c348944aa2b19e00bc2a319bcd40cbddaea9107a0b91d514c458265db39aa"} Sep 16 17:25:13.734897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.734878 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p4kfb" event={"ID":"e3a95030-580d-4d9f-b746-3359da873540","Type":"ContainerStarted","Data":"94161797bce62c7339b62e531a520d29858f9bf083513b5f36e6046fafd00c54"} Sep 16 17:25:13.736167 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.736149 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5jxl" event={"ID":"146dffd8-7263-4546-89e9-8a9b7dcc47fb","Type":"ContainerStarted","Data":"122629c88a0913d373abecd1852386a7faf0e1ab9ac46d08b8faf3c777060ed8"} Sep 16 17:25:13.737514 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.737484 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xqfp" event={"ID":"586a3038-36d1-4c2f-8739-97338f3663ba","Type":"ContainerStarted","Data":"548e7a0d68a271e2b9e4dcb2c6cde59de66b029e1ea7e747ff2ac66062196f49"} Sep 16 17:25:13.739999 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.739983 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:25:13.740355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740333 2536 generic.go:356] "Generic (PLEG): container finished" podID="2b5cbe64-815c-4b5f-91e5-2e90eb6939c0" containerID="e7d8eb5c6678608459e4623738e9b0ce5acc81d8c54224c7d32fd0a2e3406a5f" exitCode=1 Sep 16 17:25:13.740442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740399 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"61948449095a64e9a3747f5819398a58650ae1a6d199d0bd5ef0a4ebbe38d401"} Sep 16 17:25:13.740501 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740477 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"7a9c40749569043160f89538d415d09923711496ac1b4d60e14e099af906e390"} Sep 16 17:25:13.740559 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740529 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"b869ab490caedad13a5990feb913d9e92dace41f6871bc4a590e2239c0d8cd7e"} Sep 16 17:25:13.740611 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740577 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"bf501462d38a48e11c0a9963425dafa7e1f95dcd0a08cfc687f2fc6e030f0653"} Sep 16 17:25:13.740661 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.740625 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerDied","Data":"e7d8eb5c6678608459e4623738e9b0ce5acc81d8c54224c7d32fd0a2e3406a5f"} Sep 16 17:25:13.785469 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.785409 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g5jxl" podStartSLOduration=2.285170694 podStartE2EDuration="20.78538833s" totalImagesPullingTime="18.500217636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:13.785097415 +0000 UTC m=+21.728419042" watchObservedRunningTime="2026-09-16 17:25:13.78538833 +0000 UTC m=+21.728709933" Sep 16 17:25:13.866627 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.866579 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8xqfp" podStartSLOduration=1.877917387 podStartE2EDuration="20.866565699s" totalImagesPullingTime="18.988648312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:13.825445035 +0000 UTC m=+21.768766664" watchObservedRunningTime="2026-09-16 17:25:13.866565699 +0000 UTC m=+21.809887307" Sep 16 17:25:13.891653 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.891609 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p4kfb" podStartSLOduration=2.30290586 podStartE2EDuration="20.891593479s" totalImagesPullingTime="18.588687619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:13.866624212 +0000 UTC m=+21.809945834" watchObservedRunningTime="2026-09-16 17:25:13.891593479 +0000 UTC m=+21.834915080" Sep 16 17:25:13.891810 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:13.891792 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6qms" podStartSLOduration=2.314951711 podStartE2EDuration="20.891788381s" totalImagesPullingTime="18.57683667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:13.891373733 +0000 UTC m=+21.834695344" watchObservedRunningTime="2026-09-16 17:25:13.891788381 +0000 UTC m=+21.835110014" Sep 16 17:25:14.736029 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.735973 2536 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 17:25:14.746366 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.746292 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" event={"ID":"981fabd0-06a6-4d18-8fae-181cfa9cca16","Type":"ContainerStarted","Data":"8547f3080c574c9af7846d2c59d3b44c26c0a9ce6ac923f93c56e607cf901b47"} Sep 16 17:25:14.749245 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.749218 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xnm8r" event={"ID":"f33bb888-0d54-4843-99a6-f88c642ef5aa","Type":"ContainerStarted","Data":"d3e85d6f67ce40908a0aad747da34b4c4dd5ef32b645fd98e8ad516099b2f978"} Sep 16 17:25:14.751446 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.751414 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" event={"ID":"3f07d75336aa3372e5da98f816547cbd","Type":"ContainerStarted","Data":"4500d74d63b5e30fb920f28b00e7e02d6c959c5c3bcb39d202531c1c2f6fdcda"} Sep 16 17:25:14.781515 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.781468 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xnm8r" podStartSLOduration=3.129583342 podStartE2EDuration="21.781447403s" totalImagesPullingTime="18.651864061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:14.781260895 +0000 UTC m=+22.724582515" watchObservedRunningTime="2026-09-16 17:25:14.781447403 +0000 UTC m=+22.724769005" Sep 16 17:25:14.808034 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:14.807980 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-222.ec2.internal" podStartSLOduration=21.80796181 podStartE2EDuration="21.80796181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:14.807213162 +0000 UTC m=+22.750534774" watchObservedRunningTime="2026-09-16 17:25:14.80796181 +0000 UTC m=+22.751283411" Sep 16 17:25:15.448099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.447757 2536 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:25:14.735994957Z","UUID":"ae3afa8f-f4eb-440b-921c-c36aad60a8d3","Handler":null,"Name":"","Endpoint":""} Sep 16 17:25:15.449752 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.449728 2536 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 17:25:15.449891 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.449771 2536 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 17:25:15.472083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.472057 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:15.472220 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.472151 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:15.472220 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:15.472156 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:15.472220 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.472197 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:15.472360 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:15.472271 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:15.472360 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:15.472346 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:15.757516 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.757405 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:25:15.757962 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:15.757769 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"92e0e32847138f146548a27cc1975e0714f5584e609caac0dc3133af13a482a8"} Sep 16 17:25:16.391318 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.391241 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:25:16.392001 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.391975 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:25:16.763075 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.763042 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" event={"ID":"981fabd0-06a6-4d18-8fae-181cfa9cca16","Type":"ContainerStarted","Data":"95795b3c6ca6a7061fbcd795ef140b9de82f2b6301fdb26456d388ab0c160476"} Sep 16 17:25:16.763509 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.763340 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:25:16.763948 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.763910 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p4kfb" Sep 16 17:25:16.789832 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:16.789759 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sstxn" podStartSLOduration=2.520157673 podStartE2EDuration="23.789738645s" totalImagesPullingTime="21.269580972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:16.78932324 +0000 UTC m=+24.732644852" watchObservedRunningTime="2026-09-16 17:25:16.789738645 +0000 UTC m=+24.733060248" Sep 16 17:25:17.472684 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.472647 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:17.472828 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.472647 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:17.472828 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:17.472767 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:17.472944 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:17.472854 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:17.472944 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.472647 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:17.473032 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:17.472978 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:17.769631 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.769564 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:25:17.770442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.770030 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"ff1087296fc8ecc9ed49336bb5467469ff6409d1e7381813f50d5e412ddb836d"} Sep 16 17:25:17.771378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:17.770863 2536 scope.go:117] "RemoveContainer" containerID="e7d8eb5c6678608459e4623738e9b0ce5acc81d8c54224c7d32fd0a2e3406a5f" Sep 16 17:25:18.775330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.775146 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:25:18.775756 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.775671 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" event={"ID":"2b5cbe64-815c-4b5f-91e5-2e90eb6939c0","Type":"ContainerStarted","Data":"3a8a3de64602c7314759d1b764edf282151ef96aeb246d61a48b7be979b14b5b"} Sep 16 17:25:18.775854 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.775836 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:18.775934 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.775898 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:18.775987 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.775964 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:18.777752 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.777729 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="ed5e72c81680b9d75c031528e05b7dd641a3caed05d5c83a6f6c25fddf52f746" exitCode=0 Sep 16 17:25:18.777883 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.777831 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"ed5e72c81680b9d75c031528e05b7dd641a3caed05d5c83a6f6c25fddf52f746"} Sep 16 17:25:18.791081 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.791058 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:18.791207 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.791173 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:18.877348 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:18.877295 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" podStartSLOduration=7.178659594 podStartE2EDuration="25.877279584s" totalImagesPullingTime="18.69861999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:18.823061778 +0000 UTC m=+26.766383398" watchObservedRunningTime="2026-09-16 17:25:18.877279584 +0000 UTC m=+26.820601185" Sep 16 17:25:19.472506 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.472473 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:19.472697 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.472603 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:19.472697 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.472645 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:19.472817 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.472734 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:19.472817 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.472785 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:19.472906 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.472841 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:19.643521 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.643488 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t677q"] Sep 16 17:25:19.660185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.660065 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9pjl"] Sep 16 17:25:19.665578 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.665536 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxj45"] Sep 16 17:25:19.779852 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.779824 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:19.780517 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.779857 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:19.780517 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.779943 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:19.780517 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:19.779968 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:19.780517 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.780032 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:19.780517 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:19.780153 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:20.783798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:20.783764 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="e7e9d20c77ddf4c9a8b8349f46ffdab67bb77e3e0842b365f43366e85ca55770" exitCode=0 Sep 16 17:25:20.784208 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:20.783853 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"e7e9d20c77ddf4c9a8b8349f46ffdab67bb77e3e0842b365f43366e85ca55770"} Sep 16 17:25:21.472797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:21.472622 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:21.472962 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:21.472663 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:21.472962 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:21.472871 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:21.473076 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:21.472969 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:21.473076 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:21.472663 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:21.473076 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:21.473043 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:22.789620 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:22.789586 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="4e50a25873d53725937cf295c2315b66b0134820ed19719a1a7dbbd0f21fa2a5" exitCode=0 Sep 16 17:25:22.790022 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:22.789634 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"4e50a25873d53725937cf295c2315b66b0134820ed19719a1a7dbbd0f21fa2a5"} Sep 16 17:25:23.472226 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:23.472197 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:23.472378 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:23.472202 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:23.472378 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:23.472307 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t677q" podUID="a839c16c-c00b-428c-8c7e-1b25b469967f" Sep 16 17:25:23.472482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:23.472202 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:23.472482 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:23.472385 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9pjl" podUID="e63f5ebe-c8e9-4538-b46d-85d6b3eff845" Sep 16 17:25:23.472482 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:23.472455 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:25:24.864597 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.864564 2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-222.ec2.internal" event="NodeReady" Sep 16 17:25:24.865077 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.864722 2536 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 17:25:24.970889 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.970857 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5cxdl"] Sep 16 17:25:24.972973 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.972952 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:24.976222 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.976182 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bng"] Sep 16 17:25:24.978524 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.978507 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:24.979061 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.979037 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwbrz\"" Sep 16 17:25:24.979200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.979092 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 17:25:24.979283 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.979206 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 17:25:24.985843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.985663 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 17:25:24.985843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.985716 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jnc4m\"" Sep 16 17:25:24.986260 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.986240 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 17:25:24.986785 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.986765 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 17:25:24.995280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:24.995258 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bng"] Sep 16 17:25:25.000054 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.000024 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5cxdl"] Sep 16 17:25:25.041725 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.041694 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c081013d-68e2-4d33-b424-f730a3aa56b3-tmp-dir\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.041900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.041775 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9z4\" (UniqueName: \"kubernetes.io/projected/c081013d-68e2-4d33-b424-f730a3aa56b3-kube-api-access-8j9z4\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.041900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.041861 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.042043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.041935 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c081013d-68e2-4d33-b424-f730a3aa56b3-config-volume\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.142662 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142546 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c081013d-68e2-4d33-b424-f730a3aa56b3-tmp-dir\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.142662 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142612 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9z4\" (UniqueName: \"kubernetes.io/projected/c081013d-68e2-4d33-b424-f730a3aa56b3-kube-api-access-8j9z4\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.142662 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142646 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ece61df-cba2-413b-b4bd-ca48cbef29b4-cert\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.142949 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142669 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcflk\" (UniqueName: \"kubernetes.io/projected/9ece61df-cba2-413b-b4bd-ca48cbef29b4-kube-api-access-tcflk\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.142949 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142742 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.142949 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142805 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c081013d-68e2-4d33-b424-f730a3aa56b3-config-volume\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.142949 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.142931 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:25.143358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.142953 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c081013d-68e2-4d33-b424-f730a3aa56b3-tmp-dir\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.143358 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.143160 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:25.642972766 +0000 UTC m=+33.586294366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:25.144086 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.143736 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c081013d-68e2-4d33-b424-f730a3aa56b3-config-volume\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.160133 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.160100 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9z4\" (UniqueName: \"kubernetes.io/projected/c081013d-68e2-4d33-b424-f730a3aa56b3-kube-api-access-8j9z4\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.243421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.243381 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:25.243421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.243429 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.243453 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ece61df-cba2-413b-b4bd-ca48cbef29b4-cert\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.243480 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcflk\" (UniqueName: \"kubernetes.io/projected/9ece61df-cba2-413b-b4bd-ca48cbef29b4-kube-api-access-tcflk\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.243540 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.243567 2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.243603 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:25:57.243586858 +0000 UTC m=+65.186908452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 17:25:25.243636 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.243638 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret podName:a839c16c-c00b-428c-8c7e-1b25b469967f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:57.243620211 +0000 UTC m=+65.186941800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret") pod "global-pull-secret-syncer-t677q" (UID: "a839c16c-c00b-428c-8c7e-1b25b469967f") : object "kube-system"/"original-pull-secret" not registered Sep 16 17:25:25.246420 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.246392 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ece61df-cba2-413b-b4bd-ca48cbef29b4-cert\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.259881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.259850 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcflk\" (UniqueName: \"kubernetes.io/projected/9ece61df-cba2-413b-b4bd-ca48cbef29b4-kube-api-access-tcflk\") pod \"ingress-canary-g6bng\" (UID: \"9ece61df-cba2-413b-b4bd-ca48cbef29b4\") " pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.290257 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.290225 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6bng" Sep 16 17:25:25.344661 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.344619 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:25.344817 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.344799 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 17:25:25.344862 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.344820 2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 17:25:25.344862 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.344830 2536 projected.go:196] Error preparing data for projected volume kube-api-access-kzsrq for pod openshift-network-diagnostics/network-check-target-b9pjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:25.344948 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.344895 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq podName:e63f5ebe-c8e9-4538-b46d-85d6b3eff845 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:57.344882109 +0000 UTC m=+65.288203702 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzsrq" (UniqueName: "kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq") pod "network-check-target-b9pjl" (UID: "e63f5ebe-c8e9-4538-b46d-85d6b3eff845") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 17:25:25.472670 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.472638 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:25.472802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.472648 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:25.472902 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.472648 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:25.480366 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.480339 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 17:25:25.481681 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.481325 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 17:25:25.481681 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.481394 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxlnf\"" Sep 16 17:25:25.481681 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.481630 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhb2d\"" Sep 16 17:25:25.481873 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.481691 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 17:25:25.481946 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.481890 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 17:25:25.646538 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:25.646504 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:25.646732 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.646687 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:25.646796 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:25.646777 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:26.646756674 +0000 UTC m=+34.590078284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:26.653937 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:26.653372 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:26.653937 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:26.653597 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:26.653937 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:26.653649 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:28.653636828 +0000 UTC m=+36.596958409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:26.804243 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:26.804220 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerStarted","Data":"359572bcd3c49e485aa780975462b5fee88e972ebe37d0eff2401376b97ece52"} Sep 16 17:25:26.886494 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:26.886414 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bng"] Sep 16 17:25:26.890110 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:25:26.890071 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ece61df_cba2_413b_b4bd_ca48cbef29b4.slice/crio-1bc5af747b08c4f8579c4f4c3e9c751ff1598590a0a44cf5fd2ff8e211b2ff3e WatchSource:0}: Error finding container 1bc5af747b08c4f8579c4f4c3e9c751ff1598590a0a44cf5fd2ff8e211b2ff3e: Status 404 returned error can't find the container with id 1bc5af747b08c4f8579c4f4c3e9c751ff1598590a0a44cf5fd2ff8e211b2ff3e Sep 16 17:25:27.807872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:27.807841 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6bng" event={"ID":"9ece61df-cba2-413b-b4bd-ca48cbef29b4","Type":"ContainerStarted","Data":"1bc5af747b08c4f8579c4f4c3e9c751ff1598590a0a44cf5fd2ff8e211b2ff3e"} Sep 16 17:25:27.810946 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:27.810905 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="359572bcd3c49e485aa780975462b5fee88e972ebe37d0eff2401376b97ece52" exitCode=0 Sep 16 17:25:27.811071 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:27.810956 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"359572bcd3c49e485aa780975462b5fee88e972ebe37d0eff2401376b97ece52"} Sep 16 17:25:28.668700 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:28.668656 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:28.668889 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:28.668835 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:28.668957 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:28.668927 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:32.668896069 +0000 UTC m=+40.612217649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:28.816773 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:28.816733 2536 generic.go:356] "Generic (PLEG): container finished" podID="7e336a64-28bc-443a-9705-7a2b9da2ec59" containerID="0420e290f4f177f934705938f6c49060399e9569d9d27755fa77630e489c00fb" exitCode=0 Sep 16 17:25:28.817516 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:28.816821 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerDied","Data":"0420e290f4f177f934705938f6c49060399e9569d9d27755fa77630e489c00fb"} Sep 16 17:25:29.823000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:29.822747 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" event={"ID":"7e336a64-28bc-443a-9705-7a2b9da2ec59","Type":"ContainerStarted","Data":"51119dfbd590fbfd2b8d0cb598334ef82ec159f0614b26a79eedbf45e805b732"} Sep 16 17:25:29.824181 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:29.824156 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6bng" event={"ID":"9ece61df-cba2-413b-b4bd-ca48cbef29b4","Type":"ContainerStarted","Data":"e1254cb4286890b497840441fcecd94212b1f726f8a762c7a2b6210530d186cc"} Sep 16 17:25:29.862060 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:29.862017 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fm9qz" podStartSLOduration=8.311702099 podStartE2EDuration="36.862002066s" totalImagesPullingTime="28.550299967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:29.861483378 +0000 UTC m=+37.804804994" watchObservedRunningTime="2026-09-16 17:25:29.862002066 +0000 UTC m=+37.805323665" Sep 16 17:25:32.691863 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:32.691819 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:32.692261 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:32.691973 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:32.692261 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:32.692030 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:40.692015987 +0000 UTC m=+48.635337572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:40.743983 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:40.743943 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:40.744444 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:40.744051 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:40.744444 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:40.744105 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:56.744091257 +0000 UTC m=+64.687412838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:50.804993 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:50.804958 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ss65" Sep 16 17:25:50.833772 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:50.833723 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g6bng" podStartSLOduration=24.87960428 podStartE2EDuration="26.833709105s" totalImagesPullingTime="1.954104825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:29.896572018 +0000 UTC m=+37.839893618" watchObservedRunningTime="2026-09-16 17:25:50.833709105 +0000 UTC m=+58.777030715" Sep 16 17:25:56.751200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:56.751165 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:25:56.752030 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:56.751304 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:25:56.752030 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:56.751379 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:28.751358153 +0000 UTC m=+96.694679751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:25:57.255250 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.255210 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:25:57.255411 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.255260 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:57.258051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.258033 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 17:25:57.258144 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.258105 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 17:25:57.266477 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:57.266449 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 17:25:57.266581 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:25:57.266510 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:27:01.266494104 +0000 UTC m=+129.209815685 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : secret "metrics-daemon-secret" not found Sep 16 17:25:57.269688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.269662 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a839c16c-c00b-428c-8c7e-1b25b469967f-original-pull-secret\") pod \"global-pull-secret-syncer-t677q\" (UID: \"a839c16c-c00b-428c-8c7e-1b25b469967f\") " pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:57.284710 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.284690 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t677q" Sep 16 17:25:57.356633 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.356598 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:57.359864 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.359662 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 17:25:57.369908 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.369887 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 17:25:57.379952 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.379929 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsrq\" (UniqueName: \"kubernetes.io/projected/e63f5ebe-c8e9-4538-b46d-85d6b3eff845-kube-api-access-kzsrq\") pod \"network-check-target-b9pjl\" (UID: \"e63f5ebe-c8e9-4538-b46d-85d6b3eff845\") " pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:57.581094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.581025 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t677q"] Sep 16 17:25:57.584222 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:25:57.584190 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda839c16c_c00b_428c_8c7e_1b25b469967f.slice/crio-e04ce22560fcd3ce0ea78740d7772a2ff42f910ddf4a5607907d7bdac15bd44e WatchSource:0}: Error finding container e04ce22560fcd3ce0ea78740d7772a2ff42f910ddf4a5607907d7bdac15bd44e: Status 404 returned error can't find the container with id e04ce22560fcd3ce0ea78740d7772a2ff42f910ddf4a5607907d7bdac15bd44e Sep 16 17:25:57.602730 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.600216 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxlnf\"" Sep 16 17:25:57.608187 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.608171 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:25:57.889874 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.889794 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t677q" event={"ID":"a839c16c-c00b-428c-8c7e-1b25b469967f","Type":"ContainerStarted","Data":"e04ce22560fcd3ce0ea78740d7772a2ff42f910ddf4a5607907d7bdac15bd44e"} Sep 16 17:25:57.899681 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:57.899653 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9pjl"] Sep 16 17:25:57.902895 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:25:57.902870 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63f5ebe_c8e9_4538_b46d_85d6b3eff845.slice/crio-2c5a5c88c5b677d8ced5caa4e9e95eeba37c9c2ffcbd1751e32840a6f0eecd92 WatchSource:0}: Error finding container 2c5a5c88c5b677d8ced5caa4e9e95eeba37c9c2ffcbd1751e32840a6f0eecd92: Status 404 returned error can't find the container with id 2c5a5c88c5b677d8ced5caa4e9e95eeba37c9c2ffcbd1751e32840a6f0eecd92 Sep 16 17:25:58.895007 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:25:58.894954 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9pjl" event={"ID":"e63f5ebe-c8e9-4538-b46d-85d6b3eff845","Type":"ContainerStarted","Data":"2c5a5c88c5b677d8ced5caa4e9e95eeba37c9c2ffcbd1751e32840a6f0eecd92"} Sep 16 17:26:02.910195 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:02.910156 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t677q" event={"ID":"a839c16c-c00b-428c-8c7e-1b25b469967f","Type":"ContainerStarted","Data":"6e2326ccea2a734d4ecc390c93402ae1993062dbdb2d92cf5adb265676a154d8"} Sep 16 17:26:02.911653 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:02.911628 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9pjl" event={"ID":"e63f5ebe-c8e9-4538-b46d-85d6b3eff845","Type":"ContainerStarted","Data":"64b5b6a83cd9db8c62078baa0b6779c22d59c82d4e494625565fd278f97593c6"} Sep 16 17:26:02.911762 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:02.911672 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:26:02.926911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:02.926854 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t677q" podStartSLOduration=65.497660969 podStartE2EDuration="1m9.926839325s" totalImagesPullingTime="4.429178356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:02.926445851 +0000 UTC m=+70.869767452" watchObservedRunningTime="2026-09-16 17:26:02.926839325 +0000 UTC m=+70.870160925" Sep 16 17:26:02.942678 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:02.942627 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b9pjl" podStartSLOduration=65.835765903 podStartE2EDuration="1m9.942612298s" totalImagesPullingTime="4.106846395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:02.941994974 +0000 UTC m=+70.885316584" watchObservedRunningTime="2026-09-16 17:26:02.942612298 +0000 UTC m=+70.885933899" Sep 16 17:26:28.763495 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:28.763371 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:26:28.763869 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:26:28.763499 2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 17:26:28.763869 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:26:28.763552 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls podName:c081013d-68e2-4d33-b424-f730a3aa56b3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:32.763538227 +0000 UTC m=+160.706859806 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls") pod "dns-default-5cxdl" (UID: "c081013d-68e2-4d33-b424-f730a3aa56b3") : secret "dns-default-metrics-tls" not found Sep 16 17:26:33.917162 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:26:33.917132 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b9pjl" Sep 16 17:27:01.284289 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:01.284236 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:27:01.284695 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:01.284402 2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 17:27:01.284695 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:01.284497 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs podName:4cae5e35-ada9-445e-b9a6-7876dee2d9ff nodeName:}" failed. No retries permitted until 2026-09-16 17:29:03.284481546 +0000 UTC m=+251.227803126 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs") pod "network-metrics-daemon-vxj45" (UID: "4cae5e35-ada9-445e-b9a6-7876dee2d9ff") : secret "metrics-daemon-secret" not found Sep 16 17:27:11.755197 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.755165 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-fsg5h"] Sep 16 17:27:11.756741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.756726 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.759539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.759515 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 17:27:11.759671 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.759620 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 17:27:11.759671 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.759647 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 17:27:11.759782 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.759714 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 17:27:11.761224 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.761205 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6c7m\"" Sep 16 17:27:11.766506 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.766487 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 17:27:11.767150 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.767130 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fsg5h"] Sep 16 17:27:11.854886 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.854849 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-snapshots\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.855094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.854895 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-service-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.855094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.854928 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42dr\" (UniqueName: \"kubernetes.io/projected/4d026f29-dab6-48ed-b64c-83dc0670f229-kube-api-access-l42dr\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.855094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.854989 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-tmp\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.855094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.855032 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.855094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.855059 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d026f29-dab6-48ed-b64c-83dc0670f229-serving-cert\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956316 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956280 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-service-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956316 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956316 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42dr\" (UniqueName: \"kubernetes.io/projected/4d026f29-dab6-48ed-b64c-83dc0670f229-kube-api-access-l42dr\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956535 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956336 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-tmp\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956535 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956355 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956535 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956371 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d026f29-dab6-48ed-b64c-83dc0670f229-serving-cert\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956769 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-snapshots\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956861 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-tmp\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.956986 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.956960 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-service-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.957331 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.957317 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4d026f29-dab6-48ed-b64c-83dc0670f229-snapshots\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.957597 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.957580 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d026f29-dab6-48ed-b64c-83dc0670f229-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.958746 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.958725 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d026f29-dab6-48ed-b64c-83dc0670f229-serving-cert\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.969896 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.969872 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42dr\" (UniqueName: \"kubernetes.io/projected/4d026f29-dab6-48ed-b64c-83dc0670f229-kube-api-access-l42dr\") pod \"insights-operator-5844b856db-fsg5h\" (UID: \"4d026f29-dab6-48ed-b64c-83dc0670f229\") " pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:11.995386 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.995363 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd"] Sep 16 17:27:11.997107 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:11.997092 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" Sep 16 17:27:12.000711 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.000690 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69"] Sep 16 17:27:12.001808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.001790 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jjp2k\"" Sep 16 17:27:12.002334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.002313 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.002644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.002629 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 17:27:12.003074 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.003060 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 17:27:12.007980 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.007909 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 17:27:12.008063 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.008013 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 17:27:12.008652 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.008637 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 17:27:12.011467 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.011446 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd"] Sep 16 17:27:12.012445 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.012417 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 17:27:12.023146 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.023123 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69"] Sep 16 17:27:12.057363 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.057335 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvpm\" (UniqueName: \"kubernetes.io/projected/dbd8fe4e-7065-417e-ac65-ab635b34da43-kube-api-access-stvpm\") pod \"volume-data-source-validator-86695b7cc7-slxrd\" (UID: \"dbd8fe4e-7065-417e-ac65-ab635b34da43\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" Sep 16 17:27:12.057363 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.057368 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a58686-5c92-4bac-9cb1-1f11ecd81868-tmp\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.057542 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.057385 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a58686-5c92-4bac-9cb1-1f11ecd81868-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.057542 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.057409 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgvx\" (UniqueName: \"kubernetes.io/projected/63a58686-5c92-4bac-9cb1-1f11ecd81868-kube-api-access-wmgvx\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.064866 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.064836 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fsg5h" Sep 16 17:27:12.094206 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.094175 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56bccd6f44-wl52g"] Sep 16 17:27:12.095935 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.095908 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:27:12.096057 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.096035 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.097681 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.097659 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.099274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099252 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 17:27:12.099375 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099327 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 17:27:12.099375 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099356 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 17:27:12.099482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099466 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 17:27:12.099603 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099587 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 17:27:12.100021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.099999 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j862d\"" Sep 16 17:27:12.100831 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.100799 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 17:27:12.100940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.100873 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 17:27:12.101255 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.101014 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 17:27:12.101255 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.101029 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gnfmt\"" Sep 16 17:27:12.101255 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.101019 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 17:27:12.113911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.113513 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 17:27:12.122890 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.122864 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:27:12.125964 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.125942 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bccd6f44-wl52g"] Sep 16 17:27:12.158128 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158100 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158128 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158129 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-stats-auth\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.158294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158147 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7h5q\" (UniqueName: \"kubernetes.io/projected/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-kube-api-access-t7h5q\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.158294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158192 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvpm\" (UniqueName: \"kubernetes.io/projected/dbd8fe4e-7065-417e-ac65-ab635b34da43-kube-api-access-stvpm\") pod \"volume-data-source-validator-86695b7cc7-slxrd\" (UID: \"dbd8fe4e-7065-417e-ac65-ab635b34da43\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" Sep 16 17:27:12.158294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158211 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a58686-5c92-4bac-9cb1-1f11ecd81868-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.158294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158226 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a58686-5c92-4bac-9cb1-1f11ecd81868-tmp\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.158294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158252 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgvx\" (UniqueName: \"kubernetes.io/projected/63a58686-5c92-4bac-9cb1-1f11ecd81868-kube-api-access-wmgvx\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158293 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158326 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158352 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158418 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lv94\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158443 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-default-certificate\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.158482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158468 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.158712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158547 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158591 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158620 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a58686-5c92-4bac-9cb1-1f11ecd81868-tmp\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.158712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158647 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.158712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.158702 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.160688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.160660 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a58686-5c92-4bac-9cb1-1f11ecd81868-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.172295 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.172269 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvpm\" (UniqueName: \"kubernetes.io/projected/dbd8fe4e-7065-417e-ac65-ab635b34da43-kube-api-access-stvpm\") pod \"volume-data-source-validator-86695b7cc7-slxrd\" (UID: \"dbd8fe4e-7065-417e-ac65-ab635b34da43\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" Sep 16 17:27:12.173038 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.173020 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgvx\" (UniqueName: \"kubernetes.io/projected/63a58686-5c92-4bac-9cb1-1f11ecd81868-kube-api-access-wmgvx\") pod \"klusterlet-addon-workmgr-6b59596c8-jdf69\" (UID: \"63a58686-5c92-4bac-9cb1-1f11ecd81868\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.259227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259140 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259180 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259198 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.259227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259216 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259232 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-stats-auth\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.259586 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259345 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7h5q\" (UniqueName: \"kubernetes.io/projected/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-kube-api-access-t7h5q\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.259586 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.259374 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:12.759351683 +0000 UTC m=+140.702673283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:12.259586 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259454 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259586 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259507 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259586 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.259584 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.259600 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259624 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.259657 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:12.75964438 +0000 UTC m=+140.702965965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259709 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lv94\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259742 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-default-certificate\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259766 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.259826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259795 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.260179 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.259988 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.260179 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.260053 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:12.260179 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.260123 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:12.760106797 +0000 UTC m=+140.703428392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:12.260179 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.260127 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.260945 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.260901 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.262103 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.262080 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-stats-auth\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.262184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.262146 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.262184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.262153 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-default-certificate\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.262248 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.262221 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.272894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.272865 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lv94\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.273648 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.273624 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.274091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.274076 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7h5q\" (UniqueName: \"kubernetes.io/projected/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-kube-api-access-t7h5q\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.306470 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.306441 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" Sep 16 17:27:12.313086 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.313061 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:12.335043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.335016 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fsg5h"] Sep 16 17:27:12.337093 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:12.337054 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d026f29_dab6_48ed_b64c_83dc0670f229.slice/crio-7d5dcddf7972a10602ed8cfd541e971f730a0c7e76b8d08389ca5218f1869195 WatchSource:0}: Error finding container 7d5dcddf7972a10602ed8cfd541e971f730a0c7e76b8d08389ca5218f1869195: Status 404 returned error can't find the container with id 7d5dcddf7972a10602ed8cfd541e971f730a0c7e76b8d08389ca5218f1869195 Sep 16 17:27:12.580089 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.580020 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd"] Sep 16 17:27:12.582814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.582788 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69"] Sep 16 17:27:12.582970 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:12.582824 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd8fe4e_7065_417e_ac65_ab635b34da43.slice/crio-eaa3bc835f33cd96511866495e23d1cb154dc1089a207796b3cf3983465f5e6d WatchSource:0}: Error finding container eaa3bc835f33cd96511866495e23d1cb154dc1089a207796b3cf3983465f5e6d: Status 404 returned error can't find the container with id eaa3bc835f33cd96511866495e23d1cb154dc1089a207796b3cf3983465f5e6d Sep 16 17:27:12.585370 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:12.585351 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a58686_5c92_4bac_9cb1_1f11ecd81868.slice/crio-242c175b90d0eb3c18c113e15221dca6f186a9ecdc78c655dc920a52f61437ef WatchSource:0}: Error finding container 242c175b90d0eb3c18c113e15221dca6f186a9ecdc78c655dc920a52f61437ef: Status 404 returned error can't find the container with id 242c175b90d0eb3c18c113e15221dca6f186a9ecdc78c655dc920a52f61437ef Sep 16 17:27:12.764967 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.764500 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.764967 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.764603 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:12.764967 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:12.764675 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:12.764967 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.764852 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:12.764967 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.764908 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:13.764888975 +0000 UTC m=+141.708210574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:12.765593 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.765008 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:12.765593 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.765020 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:12.765939 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.765719 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:13.764947086 +0000 UTC m=+141.708268666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:12.765939 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:12.765749 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:13.765737589 +0000 UTC m=+141.709059177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:13.079873 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.079834 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" event={"ID":"dbd8fe4e-7065-417e-ac65-ab635b34da43","Type":"ContainerStarted","Data":"eaa3bc835f33cd96511866495e23d1cb154dc1089a207796b3cf3983465f5e6d"} Sep 16 17:27:13.081310 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.081278 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fsg5h" event={"ID":"4d026f29-dab6-48ed-b64c-83dc0670f229","Type":"ContainerStarted","Data":"7d5dcddf7972a10602ed8cfd541e971f730a0c7e76b8d08389ca5218f1869195"} Sep 16 17:27:13.082727 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.082695 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" event={"ID":"63a58686-5c92-4bac-9cb1-1f11ecd81868","Type":"ContainerStarted","Data":"242c175b90d0eb3c18c113e15221dca6f186a9ecdc78c655dc920a52f61437ef"} Sep 16 17:27:13.774943 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.774538 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:13.774943 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.774627 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:13.774943 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:13.774671 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:13.774943 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.774833 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.774814119 +0000 UTC m=+143.718135701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:13.775600 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.774946 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:13.775600 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.774968 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:13.775600 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.775021 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.77500434 +0000 UTC m=+143.718325926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:13.775600 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.775095 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:13.775600 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:13.775130 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.775118747 +0000 UTC m=+143.718440334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:14.087208 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:14.087172 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" event={"ID":"dbd8fe4e-7065-417e-ac65-ab635b34da43","Type":"ContainerStarted","Data":"8744a362f24215320ae5bfdd34d92f5dad702f953e816bd42e1316c268bf99a3"} Sep 16 17:27:14.107013 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:14.106948 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-slxrd" podStartSLOduration=2.005010801 podStartE2EDuration="3.106905477s" totalImagesPullingTime="1.101894676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:14.106524499 +0000 UTC m=+142.049846136" watchObservedRunningTime="2026-09-16 17:27:14.106905477 +0000 UTC m=+142.050227079" Sep 16 17:27:15.792967 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:15.792926 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:15.793024 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793056 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.793033242 +0000 UTC m=+147.736354822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793110 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793128 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:15.793149 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793169 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.793158631 +0000 UTC m=+147.736480227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793224 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:15.793434 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:15.793262 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.79324974 +0000 UTC m=+147.736571324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:16.095692 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.095601 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fsg5h" event={"ID":"4d026f29-dab6-48ed-b64c-83dc0670f229","Type":"ContainerStarted","Data":"b1f17a9836c6775eaec75e88134b868b9ffc04cff95c9f81903c2635677896d0"} Sep 16 17:27:16.723784 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.723728 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-fsg5h" podStartSLOduration=2.971201881 podStartE2EDuration="5.723709908s" totalImagesPullingTime="2.752508027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:16.113993936 +0000 UTC m=+144.057315572" watchObservedRunningTime="2026-09-16 17:27:16.723709908 +0000 UTC m=+144.667031525" Sep 16 17:27:16.724368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.724347 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v8nkj"] Sep 16 17:27:16.726341 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.726322 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.729317 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.729300 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 17:27:16.729405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.729301 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 17:27:16.729584 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.729565 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r5z4f\"" Sep 16 17:27:16.744099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.744080 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8nkj"] Sep 16 17:27:16.803314 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.803276 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef79accd-f342-4457-9cf6-e203e7ea3f2c-crio-socket\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.803897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.803345 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.803897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.803368 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.803897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.803400 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef79accd-f342-4457-9cf6-e203e7ea3f2c-data-volume\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.803897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.803473 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhc2\" (UniqueName: \"kubernetes.io/projected/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-api-access-qxhc2\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904150 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904112 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef79accd-f342-4457-9cf6-e203e7ea3f2c-data-volume\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904150 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904153 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhc2\" (UniqueName: \"kubernetes.io/projected/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-api-access-qxhc2\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904342 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904223 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef79accd-f342-4457-9cf6-e203e7ea3f2c-crio-socket\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904342 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904247 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904342 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904264 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.904432 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:16.904363 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:16.904432 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:16.904408 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:17.404395865 +0000 UTC m=+145.347717446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:16.904815 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.904797 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef79accd-f342-4457-9cf6-e203e7ea3f2c-data-volume\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.905053 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.905038 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef79accd-f342-4457-9cf6-e203e7ea3f2c-crio-socket\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:16.905097 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:16.905085 2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 17:27:16.905129 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:16.905121 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:17.40511047 +0000 UTC m=+145.348432063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : configmap "kube-rbac-proxy" not found Sep 16 17:27:16.919790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:16.919759 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhc2\" (UniqueName: \"kubernetes.io/projected/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-api-access-qxhc2\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:17.408701 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:17.408659 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:17.408886 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:17.408716 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:17.408886 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:17.408802 2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 17:27:17.408886 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:17.408835 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:17.408886 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:17.408887 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.408866197 +0000 UTC m=+146.352187797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:17.409086 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:17.408904 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.408896102 +0000 UTC m=+146.352217683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : configmap "kube-rbac-proxy" not found Sep 16 17:27:18.417864 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:18.417816 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:18.417864 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:18.417856 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:18.418423 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:18.417965 2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 17:27:18.418423 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:18.418033 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.418014748 +0000 UTC m=+148.361336341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : configmap "kube-rbac-proxy" not found Sep 16 17:27:18.418423 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:18.417972 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:18.418423 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:18.418096 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.418083417 +0000 UTC m=+148.361404999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:18.840954 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:18.840908 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5jxl_146dffd8-7263-4546-89e9-8a9b7dcc47fb/dns-node-resolver/0.log" Sep 16 17:27:19.109403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.109323 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" event={"ID":"63a58686-5c92-4bac-9cb1-1f11ecd81868","Type":"ContainerStarted","Data":"e5ca7836174c3eebd006cc12ad0e6c89db3a5ed1a95361c509738337e8b05ad1"} Sep 16 17:27:19.109597 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.109583 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:19.111292 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.111265 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" Sep 16 17:27:19.128678 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.128633 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b59596c8-jdf69" podStartSLOduration=2.464723127 podStartE2EDuration="8.128620433s" totalImagesPullingTime="5.663897306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:19.126761771 +0000 UTC m=+147.070083374" watchObservedRunningTime="2026-09-16 17:27:19.128620433 +0000 UTC m=+147.071942034" Sep 16 17:27:19.830339 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.830305 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.830379 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.830414 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830515 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830519 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830539 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830518 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.830491046 +0000 UTC m=+155.773812626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830570 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.830556461 +0000 UTC m=+155.773878042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:19.830806 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:19.830588 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.830578606 +0000 UTC m=+155.773900186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:19.840535 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:19.840514 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6qms_df81b8c0-eb31-498b-af78-a67a0cf5b8b0/node-ca/0.log" Sep 16 17:27:20.435849 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:20.435815 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:20.436038 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:20.435961 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:20.436038 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:20.435994 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:20.436111 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:20.436038 2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 17:27:20.436111 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:20.436058 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:24.436043484 +0000 UTC m=+152.379365064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:20.436111 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:20.436078 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:24.436068169 +0000 UTC m=+152.379389749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : configmap "kube-rbac-proxy" not found Sep 16 17:27:20.452702 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:20.452676 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g6bng_9ece61df-cba2-413b-b4bd-ca48cbef29b4/serve-healthcheck-canary/0.log" Sep 16 17:27:24.471418 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:24.471378 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:24.471418 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:24.471418 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:24.471826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:24.471510 2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 17:27:24.471826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:24.471543 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:24.471826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:24.471581 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:32.471565841 +0000 UTC m=+160.414887426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : configmap "kube-rbac-proxy" not found Sep 16 17:27:24.471826 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:24.471596 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:32.471590518 +0000 UTC m=+160.414912098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:27.900362 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:27.900327 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:27.900390 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:27.900415 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900485 2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900506 2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799bdcc876-pbcqs: secret "image-registry-tls" not found Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900542 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:43.900529391 +0000 UTC m=+171.843850972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : configmap references non-existent config key: service-ca.crt Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900557 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls podName:300aa621-ce46-42d8-800f-c10e08183edf nodeName:}" failed. No retries permitted until 2026-09-16 17:27:43.900550679 +0000 UTC m=+171.843872259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls") pod "image-registry-799bdcc876-pbcqs" (UID: "300aa621-ce46-42d8-800f-c10e08183edf") : secret "image-registry-tls" not found Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900556 2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 17:27:27.900707 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.900603 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs podName:3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:43.900587863 +0000 UTC m=+171.843909444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs") pod "router-default-56bccd6f44-wl52g" (UID: "3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2") : secret "router-metrics-certs-default" not found Sep 16 17:27:27.985394 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:27.985353 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5cxdl" podUID="c081013d-68e2-4d33-b424-f730a3aa56b3" Sep 16 17:27:28.135968 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:28.135938 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:28.491517 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:28.491468 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vxj45" podUID="4cae5e35-ada9-445e-b9a6-7876dee2d9ff" Sep 16 17:27:32.534256 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.534219 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:32.534701 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:32.534355 2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 17:27:32.534701 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:27:32.534411 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls podName:ef79accd-f342-4457-9cf6-e203e7ea3f2c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:48.534396072 +0000 UTC m=+176.477717657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8nkj" (UID: "ef79accd-f342-4457-9cf6-e203e7ea3f2c") : secret "insights-runtime-extractor-tls" not found Sep 16 17:27:32.534701 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.534544 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:32.535000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.534984 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef79accd-f342-4457-9cf6-e203e7ea3f2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:32.836759 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.836674 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:32.838888 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.838860 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c081013d-68e2-4d33-b424-f730a3aa56b3-metrics-tls\") pod \"dns-default-5cxdl\" (UID: \"c081013d-68e2-4d33-b424-f730a3aa56b3\") " pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:32.940288 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.940255 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwbrz\"" Sep 16 17:27:32.947785 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:32.947758 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:33.232672 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:33.232646 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5cxdl"] Sep 16 17:27:33.235484 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:33.235452 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc081013d_68e2_4d33_b424_f730a3aa56b3.slice/crio-be6415b05629b0174bc64558a6dda1ef661795ba45f08ea0893767dd60317285 WatchSource:0}: Error finding container be6415b05629b0174bc64558a6dda1ef661795ba45f08ea0893767dd60317285: Status 404 returned error can't find the container with id be6415b05629b0174bc64558a6dda1ef661795ba45f08ea0893767dd60317285 Sep 16 17:27:34.154161 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:34.154118 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cxdl" event={"ID":"c081013d-68e2-4d33-b424-f730a3aa56b3","Type":"ContainerStarted","Data":"be6415b05629b0174bc64558a6dda1ef661795ba45f08ea0893767dd60317285"} Sep 16 17:27:36.163805 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:36.163768 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cxdl" event={"ID":"c081013d-68e2-4d33-b424-f730a3aa56b3","Type":"ContainerStarted","Data":"b0c6a3d1d17c781fc6ac19d16d3bd42b0b212cd6486eaca5d4db68ae2ca72979"} Sep 16 17:27:36.164249 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:36.163843 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cxdl" event={"ID":"c081013d-68e2-4d33-b424-f730a3aa56b3","Type":"ContainerStarted","Data":"e2168ffee737508ecc1169e807103122ca1d874352a61c1e2b18e08cae4fb830"} Sep 16 17:27:36.164249 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:36.163878 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:36.182526 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:36.182472 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5cxdl" podStartSLOduration=130.394825843 podStartE2EDuration="2m12.182458399s" totalImagesPullingTime="1.787632556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:36.181078398 +0000 UTC m=+164.124400000" watchObservedRunningTime="2026-09-16 17:27:36.182458399 +0000 UTC m=+164.125780021" Sep 16 17:27:39.472182 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:39.472144 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:27:43.922985 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.922949 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:43.923355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.923002 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:43.923355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.923030 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:43.923558 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.923539 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-service-ca-bundle\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:43.925253 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.925233 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2-metrics-certs\") pod \"router-default-56bccd6f44-wl52g\" (UID: \"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2\") " pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:43.925345 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:43.925327 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"image-registry-799bdcc876-pbcqs\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:44.207112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.207031 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:44.213079 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.213055 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:44.492982 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.492955 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bccd6f44-wl52g"] Sep 16 17:27:44.494054 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:44.494028 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8d0ef0_3c08_4c17_b45e_fb58983cbeb2.slice/crio-481970c37811e367d7e89a89c57e33d4f28e29b9479d062b0e83fef64b41b7bc WatchSource:0}: Error finding container 481970c37811e367d7e89a89c57e33d4f28e29b9479d062b0e83fef64b41b7bc: Status 404 returned error can't find the container with id 481970c37811e367d7e89a89c57e33d4f28e29b9479d062b0e83fef64b41b7bc Sep 16 17:27:44.533404 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.533380 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:27:44.536688 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:44.536661 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300aa621_ce46_42d8_800f_c10e08183edf.slice/crio-d94e3a5da165dc7a9fc7f36cd1210a940dc5dd755cc068423841041e2a184ad0 WatchSource:0}: Error finding container d94e3a5da165dc7a9fc7f36cd1210a940dc5dd755cc068423841041e2a184ad0: Status 404 returned error can't find the container with id d94e3a5da165dc7a9fc7f36cd1210a940dc5dd755cc068423841041e2a184ad0 Sep 16 17:27:44.609329 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.609305 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:27:44.699113 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.699074 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f499fd44b-rfrwj"] Sep 16 17:27:44.701297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.701269 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.720793 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.720766 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f499fd44b-rfrwj"] Sep 16 17:27:44.728010 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.727985 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-image-registry-private-configuration\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728020 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-trusted-ca\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728055 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-bound-sa-token\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728093 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-installation-pull-secrets\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728116 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e601eaf-2fbe-40bb-91e1-376e6f635338-ca-trust-extracted\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728159 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728139 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4m2q\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-kube-api-access-g4m2q\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728372 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728189 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-certificates\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.728372 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.728272 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-tls\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829145 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829061 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-bound-sa-token\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829145 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829098 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-installation-pull-secrets\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829145 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829118 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e601eaf-2fbe-40bb-91e1-376e6f635338-ca-trust-extracted\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829145 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829144 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4m2q\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-kube-api-access-g4m2q\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829205 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-certificates\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829307 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-tls\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829343 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-image-registry-private-configuration\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829391 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-trusted-ca\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.829880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.829854 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e601eaf-2fbe-40bb-91e1-376e6f635338-ca-trust-extracted\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.830130 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.830109 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-certificates\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.830715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.830691 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e601eaf-2fbe-40bb-91e1-376e6f635338-trusted-ca\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.831731 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.831712 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-registry-tls\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.831940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.831894 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-image-registry-private-configuration\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.832061 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.832041 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e601eaf-2fbe-40bb-91e1-376e6f635338-installation-pull-secrets\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.843907 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.843888 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4m2q\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-kube-api-access-g4m2q\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:44.848581 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:44.848564 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e601eaf-2fbe-40bb-91e1-376e6f635338-bound-sa-token\") pod \"image-registry-5f499fd44b-rfrwj\" (UID: \"2e601eaf-2fbe-40bb-91e1-376e6f635338\") " pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:45.011728 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.011648 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:45.193704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.193627 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" event={"ID":"300aa621-ce46-42d8-800f-c10e08183edf","Type":"ContainerStarted","Data":"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c"} Sep 16 17:27:45.193704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.193675 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" event={"ID":"300aa621-ce46-42d8-800f-c10e08183edf","Type":"ContainerStarted","Data":"d94e3a5da165dc7a9fc7f36cd1210a940dc5dd755cc068423841041e2a184ad0"} Sep 16 17:27:45.194027 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.193716 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:27:45.195184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.195161 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bccd6f44-wl52g" event={"ID":"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2","Type":"ContainerStarted","Data":"11f3ddb4a4f30803160e320f18d9bbfff253d0df49b84da826b681038310e41b"} Sep 16 17:27:45.195281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.195203 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bccd6f44-wl52g" event={"ID":"3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2","Type":"ContainerStarted","Data":"481970c37811e367d7e89a89c57e33d4f28e29b9479d062b0e83fef64b41b7bc"} Sep 16 17:27:45.207513 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.207494 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:45.209796 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.209748 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:45.313166 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.313116 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" podStartSLOduration=33.313097649 podStartE2EDuration="33.313097649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:45.268264032 +0000 UTC m=+173.211585634" watchObservedRunningTime="2026-09-16 17:27:45.313097649 +0000 UTC m=+173.256419250" Sep 16 17:27:45.314321 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:45.314296 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f499fd44b-rfrwj"] Sep 16 17:27:45.316835 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:45.316810 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e601eaf_2fbe_40bb_91e1_376e6f635338.slice/crio-9b51696eb9803a00b85c28617d6551d4801a9db4cb1b0f74b7c6e2c0c4824037 WatchSource:0}: Error finding container 9b51696eb9803a00b85c28617d6551d4801a9db4cb1b0f74b7c6e2c0c4824037: Status 404 returned error can't find the container with id 9b51696eb9803a00b85c28617d6551d4801a9db4cb1b0f74b7c6e2c0c4824037 Sep 16 17:27:46.168650 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.168621 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5cxdl" Sep 16 17:27:46.192177 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.192129 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56bccd6f44-wl52g" podStartSLOduration=34.192109118 podStartE2EDuration="34.192109118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:45.328577812 +0000 UTC m=+173.271899425" watchObservedRunningTime="2026-09-16 17:27:46.192109118 +0000 UTC m=+174.135430723" Sep 16 17:27:46.199843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.199811 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" event={"ID":"2e601eaf-2fbe-40bb-91e1-376e6f635338","Type":"ContainerStarted","Data":"bfbdccbe4f592465f7c49c09e1aa71e1fe90fb2dde9e9780992e3795c475e10f"} Sep 16 17:27:46.200022 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.199871 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" event={"ID":"2e601eaf-2fbe-40bb-91e1-376e6f635338","Type":"ContainerStarted","Data":"9b51696eb9803a00b85c28617d6551d4801a9db4cb1b0f74b7c6e2c0c4824037"} Sep 16 17:27:46.200022 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.199953 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:27:46.200126 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.200098 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:46.201334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.201315 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56bccd6f44-wl52g" Sep 16 17:27:46.234136 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:46.234088 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" podStartSLOduration=2.234073216 podStartE2EDuration="2.234073216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:46.233314236 +0000 UTC m=+174.176635869" watchObservedRunningTime="2026-09-16 17:27:46.234073216 +0000 UTC m=+174.177394818" Sep 16 17:27:48.559627 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:48.559586 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:48.562025 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:48.561999 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef79accd-f342-4457-9cf6-e203e7ea3f2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8nkj\" (UID: \"ef79accd-f342-4457-9cf6-e203e7ea3f2c\") " pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:48.835546 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:48.835468 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8nkj" Sep 16 17:27:49.114558 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:49.114467 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8nkj"] Sep 16 17:27:49.118978 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:49.118943 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef79accd_f342_4457_9cf6_e203e7ea3f2c.slice/crio-f7233fcfc969c4c462b53e7bbbd55efc58ce866d348a0c7f25788f95141bfd70 WatchSource:0}: Error finding container f7233fcfc969c4c462b53e7bbbd55efc58ce866d348a0c7f25788f95141bfd70: Status 404 returned error can't find the container with id f7233fcfc969c4c462b53e7bbbd55efc58ce866d348a0c7f25788f95141bfd70 Sep 16 17:27:49.210703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:49.210677 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8nkj" event={"ID":"ef79accd-f342-4457-9cf6-e203e7ea3f2c","Type":"ContainerStarted","Data":"9e7a1c808b31513a5cc50fc39f4796e58bd100d33bd3032e2771094753fdebd3"} Sep 16 17:27:49.210843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:49.210727 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8nkj" event={"ID":"ef79accd-f342-4457-9cf6-e203e7ea3f2c","Type":"ContainerStarted","Data":"f7233fcfc969c4c462b53e7bbbd55efc58ce866d348a0c7f25788f95141bfd70"} Sep 16 17:27:50.215891 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:50.215858 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8nkj" event={"ID":"ef79accd-f342-4457-9cf6-e203e7ea3f2c","Type":"ContainerStarted","Data":"4ea868eb9d8efd14abdf2f93ccfed0a6c095e89239864887cb78844d91db3b02"} Sep 16 17:27:51.221633 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:51.221602 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8nkj" event={"ID":"ef79accd-f342-4457-9cf6-e203e7ea3f2c","Type":"ContainerStarted","Data":"cdf5315d3e1eb488a112f7c8fa7a4d71d895c935dec5c3e51215fc1d60eb0a76"} Sep 16 17:27:51.249677 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:51.249632 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v8nkj" podStartSLOduration=33.386054666 podStartE2EDuration="35.249615215s" totalImagesPullingTime="1.863560549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:51.24888104 +0000 UTC m=+179.192202652" watchObservedRunningTime="2026-09-16 17:27:51.249615215 +0000 UTC m=+179.192936816" Sep 16 17:27:54.250440 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.250408 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nzk4k"] Sep 16 17:27:54.253030 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.253014 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.259706 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.259683 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 17:27:54.259802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.259745 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 17:27:54.259802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.259776 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 17:27:54.261640 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.261623 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 17:27:54.261733 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.261710 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 17:27:54.261785 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.261769 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 17:27:54.261833 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.261766 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nv8hc\"" Sep 16 17:27:54.303880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.303845 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-tls\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.303880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.303880 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6m2\" (UniqueName: \"kubernetes.io/projected/e6e500d3-c03a-4a33-a1a2-c881d271cd84-kube-api-access-fh6m2\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.303908 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-sys\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.303962 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-root\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.303991 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-metrics-client-ca\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.304063 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-textfile\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.304090 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-wtmp\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304112 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.304109 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.304289 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.304176 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405449 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405394 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-root\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405449 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405453 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-metrics-client-ca\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405484 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-textfile\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405501 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-wtmp\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405510 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-root\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405518 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405572 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405623 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-tls\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405645 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6m2\" (UniqueName: \"kubernetes.io/projected/e6e500d3-c03a-4a33-a1a2-c881d271cd84-kube-api-access-fh6m2\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405677 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-sys\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.405698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405695 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-wtmp\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.406078 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405755 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6e500d3-c03a-4a33-a1a2-c881d271cd84-sys\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.406078 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.405864 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-textfile\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.406183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.406164 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-metrics-client-ca\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.406183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.406175 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.408009 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.407989 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.408077 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.408023 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6e500d3-c03a-4a33-a1a2-c881d271cd84-node-exporter-tls\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.436614 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.436577 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6m2\" (UniqueName: \"kubernetes.io/projected/e6e500d3-c03a-4a33-a1a2-c881d271cd84-kube-api-access-fh6m2\") pod \"node-exporter-nzk4k\" (UID: \"e6e500d3-c03a-4a33-a1a2-c881d271cd84\") " pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.561856 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:54.561764 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzk4k" Sep 16 17:27:54.570854 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:54.570830 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e500d3_c03a_4a33_a1a2_c881d271cd84.slice/crio-26c1cd72bcd95dbf97a76ed309daaf4b9f04afd22e27e5fd0744777af3f100d6 WatchSource:0}: Error finding container 26c1cd72bcd95dbf97a76ed309daaf4b9f04afd22e27e5fd0744777af3f100d6: Status 404 returned error can't find the container with id 26c1cd72bcd95dbf97a76ed309daaf4b9f04afd22e27e5fd0744777af3f100d6 Sep 16 17:27:55.238740 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:55.238705 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzk4k" event={"ID":"e6e500d3-c03a-4a33-a1a2-c881d271cd84","Type":"ContainerStarted","Data":"26c1cd72bcd95dbf97a76ed309daaf4b9f04afd22e27e5fd0744777af3f100d6"} Sep 16 17:27:56.243874 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:56.243838 2536 generic.go:356] "Generic (PLEG): container finished" podID="e6e500d3-c03a-4a33-a1a2-c881d271cd84" containerID="992f13479d8656a5a67b7a657b3d7cfe75513d783afe9e35e9562d7bd9ad2f45" exitCode=0 Sep 16 17:27:56.244235 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:56.243935 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzk4k" event={"ID":"e6e500d3-c03a-4a33-a1a2-c881d271cd84","Type":"ContainerDied","Data":"992f13479d8656a5a67b7a657b3d7cfe75513d783afe9e35e9562d7bd9ad2f45"} Sep 16 17:27:57.250179 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:57.250144 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzk4k" event={"ID":"e6e500d3-c03a-4a33-a1a2-c881d271cd84","Type":"ContainerStarted","Data":"7c0647e683e0fef90d92ba4187985cce06f8dfa5ce675d2d08e0bf711b36ad5e"} Sep 16 17:27:57.250555 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:57.250205 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzk4k" event={"ID":"e6e500d3-c03a-4a33-a1a2-c881d271cd84","Type":"ContainerStarted","Data":"d47cc0bb40670a73b1ff4f8a91f23340d6be849240a6e76649a39c28f3ffb21e"} Sep 16 17:27:57.291310 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:57.291261 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nzk4k" podStartSLOduration=2.04564287 podStartE2EDuration="3.29124426s" totalImagesPullingTime="1.183458858s" totalInitContainerRuntime="62.142532ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:57.289699997 +0000 UTC m=+185.233021618" watchObservedRunningTime="2026-09-16 17:27:57.29124426 +0000 UTC m=+185.234565860" Sep 16 17:27:58.864726 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.864693 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6858ddbdb-cxvw7"] Sep 16 17:27:58.866640 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.866624 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.871662 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.871627 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 17:27:58.872747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.872677 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 17:27:58.872894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.872873 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xxwwp\"" Sep 16 17:27:58.872894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.872873 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-hqm3f0cdltu3\"" Sep 16 17:27:58.873047 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.872911 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 17:27:58.873362 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.873344 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 17:27:58.883490 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.883469 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6858ddbdb-cxvw7"] Sep 16 17:27:58.908308 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.908276 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92"] Sep 16 17:27:58.910120 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.910100 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:58.912661 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.912639 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lwddn\"" Sep 16 17:27:58.912732 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.912685 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 17:27:58.920726 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.920703 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92"] Sep 16 17:27:58.946637 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946610 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-tls\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946768 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946661 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-metrics-server-audit-profiles\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946768 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946687 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-client-certs\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946768 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946701 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6vj\" (UniqueName: \"kubernetes.io/projected/61a5a0dd-93a3-42b8-8332-4798a0728698-kube-api-access-hj6vj\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946768 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946748 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61a5a0dd-93a3-42b8-8332-4798a0728698-audit-log\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946769 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:58.946894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:58.946787 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-client-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047559 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61a5a0dd-93a3-42b8-8332-4798a0728698-audit-log\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047595 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047616 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-client-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047740 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-tls\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047788 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90aae6d6-be66-435d-a79a-e2115d70ca1b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.047993 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047818 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7bq\" (UniqueName: \"kubernetes.io/projected/90aae6d6-be66-435d-a79a-e2115d70ca1b-kube-api-access-kl7bq\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.047993 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047890 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-metrics-server-audit-profiles\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.047993 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.047974 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-client-certs\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.048137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.048008 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6vj\" (UniqueName: \"kubernetes.io/projected/61a5a0dd-93a3-42b8-8332-4798a0728698-kube-api-access-hj6vj\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.048137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.048091 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61a5a0dd-93a3-42b8-8332-4798a0728698-audit-log\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.048512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.048487 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.048880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.048854 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61a5a0dd-93a3-42b8-8332-4798a0728698-metrics-server-audit-profiles\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.050172 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.050152 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-tls\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.050303 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.050283 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-client-ca-bundle\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.050653 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.050635 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61a5a0dd-93a3-42b8-8332-4798a0728698-secret-metrics-server-client-certs\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.065698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.065674 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6vj\" (UniqueName: \"kubernetes.io/projected/61a5a0dd-93a3-42b8-8332-4798a0728698-kube-api-access-hj6vj\") pod \"metrics-server-6858ddbdb-cxvw7\" (UID: \"61a5a0dd-93a3-42b8-8332-4798a0728698\") " pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.148520 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.148447 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90aae6d6-be66-435d-a79a-e2115d70ca1b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.148520 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.148486 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7bq\" (UniqueName: \"kubernetes.io/projected/90aae6d6-be66-435d-a79a-e2115d70ca1b-kube-api-access-kl7bq\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.150910 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.150881 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90aae6d6-be66-435d-a79a-e2115d70ca1b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.163686 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.163659 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7bq\" (UniqueName: \"kubernetes.io/projected/90aae6d6-be66-435d-a79a-e2115d70ca1b-kube-api-access-kl7bq\") pod \"monitoring-plugin-5d85778b9d-mzs92\" (UID: \"90aae6d6-be66-435d-a79a-e2115d70ca1b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.175555 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.175532 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:27:59.218343 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.218175 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:27:59.446064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.446039 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6858ddbdb-cxvw7"] Sep 16 17:27:59.448392 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:59.448361 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a5a0dd_93a3_42b8_8332_4798a0728698.slice/crio-a52aaae92a1a33b54f895f7f0c267f364ab189d4abd87c71da498c27d8d3a560 WatchSource:0}: Error finding container a52aaae92a1a33b54f895f7f0c267f364ab189d4abd87c71da498c27d8d3a560: Status 404 returned error can't find the container with id a52aaae92a1a33b54f895f7f0c267f364ab189d4abd87c71da498c27d8d3a560 Sep 16 17:27:59.494008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:27:59.493973 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92"] Sep 16 17:27:59.496934 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:27:59.496895 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aae6d6_be66_435d_a79a_e2115d70ca1b.slice/crio-894c0f1fca121773d0db6ebb3a59f58cfc0cf68a55b62c49d2ac50a32dfb8ac0 WatchSource:0}: Error finding container 894c0f1fca121773d0db6ebb3a59f58cfc0cf68a55b62c49d2ac50a32dfb8ac0: Status 404 returned error can't find the container with id 894c0f1fca121773d0db6ebb3a59f58cfc0cf68a55b62c49d2ac50a32dfb8ac0 Sep 16 17:28:00.265302 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:00.265260 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" event={"ID":"61a5a0dd-93a3-42b8-8332-4798a0728698","Type":"ContainerStarted","Data":"a52aaae92a1a33b54f895f7f0c267f364ab189d4abd87c71da498c27d8d3a560"} Sep 16 17:28:00.266596 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:00.266562 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" event={"ID":"90aae6d6-be66-435d-a79a-e2115d70ca1b","Type":"ContainerStarted","Data":"894c0f1fca121773d0db6ebb3a59f58cfc0cf68a55b62c49d2ac50a32dfb8ac0"} Sep 16 17:28:02.275506 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.275473 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" event={"ID":"61a5a0dd-93a3-42b8-8332-4798a0728698","Type":"ContainerStarted","Data":"816f269b16cd03a52988d6c876fcf66618a314666239bcef6bf12883e9979376"} Sep 16 17:28:02.277008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.276987 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" event={"ID":"90aae6d6-be66-435d-a79a-e2115d70ca1b","Type":"ContainerStarted","Data":"454c2e66a7d25de3967db24b6941c1b07bafe3859b83a433f21f92dd9ced0ef4"} Sep 16 17:28:02.277121 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.277103 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:28:02.281790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.281772 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" Sep 16 17:28:02.296548 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.296501 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" podStartSLOduration=2.51934361 podStartE2EDuration="4.296486588s" totalImagesPullingTime="1.777142978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:02.295053747 +0000 UTC m=+190.238375360" watchObservedRunningTime="2026-09-16 17:28:02.296486588 +0000 UTC m=+190.239808190" Sep 16 17:28:02.315204 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:02.315159 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mzs92" podStartSLOduration=2.5879835780000002 podStartE2EDuration="4.315143326s" totalImagesPullingTime="1.727159748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:02.313526579 +0000 UTC m=+190.256848200" watchObservedRunningTime="2026-09-16 17:28:02.315143326 +0000 UTC m=+190.258464928" Sep 16 17:28:06.205101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.205074 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:28:06.828974 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.828936 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-p5tkm"] Sep 16 17:28:06.831672 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.831650 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:06.834757 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.834733 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 17:28:06.835145 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.835130 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 17:28:06.836104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.836088 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cpsnn\"" Sep 16 17:28:06.845840 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.845820 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-p5tkm"] Sep 16 17:28:06.916493 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:06.916453 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkztg\" (UniqueName: \"kubernetes.io/projected/777ee835-a9c7-4eaa-b8ec-53db29935526-kube-api-access-wkztg\") pod \"downloads-76f99584f4-p5tkm\" (UID: \"777ee835-a9c7-4eaa-b8ec-53db29935526\") " pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:07.017320 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:07.017288 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkztg\" (UniqueName: \"kubernetes.io/projected/777ee835-a9c7-4eaa-b8ec-53db29935526-kube-api-access-wkztg\") pod \"downloads-76f99584f4-p5tkm\" (UID: \"777ee835-a9c7-4eaa-b8ec-53db29935526\") " pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:07.025561 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:07.025530 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkztg\" (UniqueName: \"kubernetes.io/projected/777ee835-a9c7-4eaa-b8ec-53db29935526-kube-api-access-wkztg\") pod \"downloads-76f99584f4-p5tkm\" (UID: \"777ee835-a9c7-4eaa-b8ec-53db29935526\") " pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:07.140405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:07.140316 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:07.208563 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:07.208533 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f499fd44b-rfrwj" Sep 16 17:28:07.413031 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:07.412961 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-p5tkm"] Sep 16 17:28:07.416415 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:28:07.416390 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777ee835_a9c7_4eaa_b8ec_53db29935526.slice/crio-b3ce61b7b756bf2ecd9874c1a1e7e366838223222ce7f593a486bdf3a0755bef WatchSource:0}: Error finding container b3ce61b7b756bf2ecd9874c1a1e7e366838223222ce7f593a486bdf3a0755bef: Status 404 returned error can't find the container with id b3ce61b7b756bf2ecd9874c1a1e7e366838223222ce7f593a486bdf3a0755bef Sep 16 17:28:08.298641 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:08.298602 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-p5tkm" event={"ID":"777ee835-a9c7-4eaa-b8ec-53db29935526","Type":"ContainerStarted","Data":"b3ce61b7b756bf2ecd9874c1a1e7e366838223222ce7f593a486bdf3a0755bef"} Sep 16 17:28:11.219107 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.219045 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" podUID="300aa621-ce46-42d8-800f-c10e08183edf" containerName="registry" containerID="cri-o://b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c" gracePeriod=30 Sep 16 17:28:11.455105 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.455080 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:28:11.562117 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562033 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562117 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562087 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562344 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562118 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562344 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562145 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562592 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562546 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562722 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562597 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562722 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562659 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.562722 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.562690 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lv94\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94\") pod \"300aa621-ce46-42d8-800f-c10e08183edf\" (UID: \"300aa621-ce46-42d8-800f-c10e08183edf\") " Sep 16 17:28:11.563046 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.563015 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:28:11.563317 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.563284 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:28:11.564704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.564679 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:28:11.565226 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.565182 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94" (OuterVolumeSpecName: "kube-api-access-2lv94") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "kube-api-access-2lv94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:28:11.565321 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.565251 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:28:11.565405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.565382 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:28:11.566980 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.566945 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:28:11.571977 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.571954 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "300aa621-ce46-42d8-800f-c10e08183edf" (UID: "300aa621-ce46-42d8-800f-c10e08183edf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:28:11.664416 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664382 2536 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300aa621-ce46-42d8-800f-c10e08183edf-ca-trust-extracted\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664416 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664411 2536 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-bound-sa-token\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664416 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664421 2536 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-installation-pull-secrets\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664434 2536 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300aa621-ce46-42d8-800f-c10e08183edf-image-registry-private-configuration\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664448 2536 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-trusted-ca\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664456 2536 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300aa621-ce46-42d8-800f-c10e08183edf-registry-certificates\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664463 2536 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-registry-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:11.664644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:11.664471 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lv94\" (UniqueName: \"kubernetes.io/projected/300aa621-ce46-42d8-800f-c10e08183edf-kube-api-access-2lv94\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:28:12.314277 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.314245 2536 generic.go:356] "Generic (PLEG): container finished" podID="300aa621-ce46-42d8-800f-c10e08183edf" containerID="b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c" exitCode=0 Sep 16 17:28:12.314717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.314291 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" Sep 16 17:28:12.314717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.314292 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" event={"ID":"300aa621-ce46-42d8-800f-c10e08183edf","Type":"ContainerDied","Data":"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c"} Sep 16 17:28:12.314717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.314424 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799bdcc876-pbcqs" event={"ID":"300aa621-ce46-42d8-800f-c10e08183edf","Type":"ContainerDied","Data":"d94e3a5da165dc7a9fc7f36cd1210a940dc5dd755cc068423841041e2a184ad0"} Sep 16 17:28:12.314717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.314469 2536 scope.go:117] "RemoveContainer" containerID="b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c" Sep 16 17:28:12.324403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.324377 2536 scope.go:117] "RemoveContainer" containerID="b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c" Sep 16 17:28:12.324771 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:28:12.324743 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c\": container with ID starting with b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c not found: ID does not exist" containerID="b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c" Sep 16 17:28:12.324862 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.324783 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c"} err="failed to get container status \"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c\": rpc error: code = NotFound desc = could not find container \"b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c\": container with ID starting with b82c0115e2dc2bfbe834976ddade9c25fc0d7dc9fe49c62726496524c219c22c not found: ID does not exist" Sep 16 17:28:12.345896 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.345868 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:28:12.349871 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.349845 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799bdcc876-pbcqs"] Sep 16 17:28:12.477442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:12.477414 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="300aa621-ce46-42d8-800f-c10e08183edf" path="/var/lib/kubelet/pods/300aa621-ce46-42d8-800f-c10e08183edf/volumes" Sep 16 17:28:17.197301 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.197259 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:28:17.197715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.197547 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300aa621-ce46-42d8-800f-c10e08183edf" containerName="registry" Sep 16 17:28:17.197715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.197561 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="300aa621-ce46-42d8-800f-c10e08183edf" containerName="registry" Sep 16 17:28:17.197715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.197629 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="300aa621-ce46-42d8-800f-c10e08183edf" containerName="registry" Sep 16 17:28:17.199925 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.199892 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.203357 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.203334 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 17:28:17.203475 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.203409 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 17:28:17.204135 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.204116 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 17:28:17.204848 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.204829 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 17:28:17.205130 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.205112 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 17:28:17.206084 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.206034 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j2r4f\"" Sep 16 17:28:17.217550 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.217528 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:28:17.313251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313205 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.313418 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313258 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.313418 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313370 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wwp\" (UniqueName: \"kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.313519 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313434 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.313519 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313478 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.313519 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.313503 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414439 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414399 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wwp\" (UniqueName: \"kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414460 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414504 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414523 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414574 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.414615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.414609 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.415432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.415373 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.415432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.415394 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.415615 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.415470 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.417380 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.417353 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.417592 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.417573 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.423388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.423364 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wwp\" (UniqueName: \"kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp\") pod \"console-679b59c599-x4r4h\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:17.510190 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:17.510155 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:19.176582 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:19.176544 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:28:19.177062 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:19.176614 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:28:22.876244 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:22.876155 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:28:22.880636 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:28:22.880604 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5221f7_ea01_437d_bcb8_a575165c1ffa.slice/crio-3e61c8028810bddef0112fcce5798b26baaf1f4a385c68046a16db8db47fef69 WatchSource:0}: Error finding container 3e61c8028810bddef0112fcce5798b26baaf1f4a385c68046a16db8db47fef69: Status 404 returned error can't find the container with id 3e61c8028810bddef0112fcce5798b26baaf1f4a385c68046a16db8db47fef69 Sep 16 17:28:23.367358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:23.367308 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-p5tkm" event={"ID":"777ee835-a9c7-4eaa-b8ec-53db29935526","Type":"ContainerStarted","Data":"e07172bc929bf134071ffc915981b9ad026b2a0b6a4bceb1709fa9bf1218f12b"} Sep 16 17:28:23.367546 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:23.367420 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:23.368954 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:23.368904 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b59c599-x4r4h" event={"ID":"2f5221f7-ea01-437d-bcb8-a575165c1ffa","Type":"ContainerStarted","Data":"3e61c8028810bddef0112fcce5798b26baaf1f4a385c68046a16db8db47fef69"} Sep 16 17:28:23.380766 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:23.380743 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-p5tkm" Sep 16 17:28:23.386963 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:23.386897 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-p5tkm" podStartSLOduration=2.122821644 podStartE2EDuration="17.38688195s" totalImagesPullingTime="15.264060306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:23.38579284 +0000 UTC m=+211.329114468" watchObservedRunningTime="2026-09-16 17:28:23.38688195 +0000 UTC m=+211.330203551" Sep 16 17:28:24.914558 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.914518 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:28:24.951746 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.951709 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:28:24.951999 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.951852 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.962471 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.962163 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 17:28:24.975953 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.975898 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.975988 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.976026 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.976053 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976323 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.976109 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9w2l\" (UniqueName: \"kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976323 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.976160 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:24.976323 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:24.976231 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077543 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077475 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077543 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077531 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077562 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077590 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077629 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9w2l\" (UniqueName: \"kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077678 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.077789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.077718 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.078636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.078593 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.078866 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.078825 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.079095 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.079047 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.079200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.079160 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.082163 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.082140 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.082356 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.082166 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.087872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.087789 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9w2l\" (UniqueName: \"kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l\") pod \"console-646b9ff67-zzs7r\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:25.266679 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:25.266540 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:26.725376 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:26.725343 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:28:26.821079 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:28:26.821037 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41be4528_44f6_4221_be83_dbc721b2384b.slice/crio-828a85b8d654021a404eb11751a24ace76afff8794c44bf6e5cce9a9bb945a31 WatchSource:0}: Error finding container 828a85b8d654021a404eb11751a24ace76afff8794c44bf6e5cce9a9bb945a31: Status 404 returned error can't find the container with id 828a85b8d654021a404eb11751a24ace76afff8794c44bf6e5cce9a9bb945a31 Sep 16 17:28:27.387767 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.387728 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b59c599-x4r4h" event={"ID":"2f5221f7-ea01-437d-bcb8-a575165c1ffa","Type":"ContainerStarted","Data":"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a"} Sep 16 17:28:27.389815 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.389788 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646b9ff67-zzs7r" event={"ID":"41be4528-44f6-4221-be83-dbc721b2384b","Type":"ContainerStarted","Data":"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f"} Sep 16 17:28:27.389976 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.389833 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646b9ff67-zzs7r" event={"ID":"41be4528-44f6-4221-be83-dbc721b2384b","Type":"ContainerStarted","Data":"828a85b8d654021a404eb11751a24ace76afff8794c44bf6e5cce9a9bb945a31"} Sep 16 17:28:27.409347 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.409295 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679b59c599-x4r4h" podStartSLOduration=6.447971995 podStartE2EDuration="10.40927662s" totalImagesPullingTime="3.961304625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:27.407316982 +0000 UTC m=+215.350638619" watchObservedRunningTime="2026-09-16 17:28:27.40927662 +0000 UTC m=+215.352598283" Sep 16 17:28:27.425629 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.425570 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-646b9ff67-zzs7r" podStartSLOduration=3.425549952 podStartE2EDuration="3.425549952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:27.42490821 +0000 UTC m=+215.368229841" watchObservedRunningTime="2026-09-16 17:28:27.425549952 +0000 UTC m=+215.368871554" Sep 16 17:28:27.511302 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.511270 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:27.511484 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.511327 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:27.516650 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:27.516626 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:28.397837 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:28.397803 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:28:35.267770 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:35.267731 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:35.268348 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:35.267799 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:35.272445 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:35.272420 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:35.423689 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:35.423659 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:28:35.469814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:35.469784 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:28:36.425156 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:36.425071 2536 generic.go:356] "Generic (PLEG): container finished" podID="4d026f29-dab6-48ed-b64c-83dc0670f229" containerID="b1f17a9836c6775eaec75e88134b868b9ffc04cff95c9f81903c2635677896d0" exitCode=0 Sep 16 17:28:36.425577 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:36.425147 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fsg5h" event={"ID":"4d026f29-dab6-48ed-b64c-83dc0670f229","Type":"ContainerDied","Data":"b1f17a9836c6775eaec75e88134b868b9ffc04cff95c9f81903c2635677896d0"} Sep 16 17:28:36.425577 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:36.425495 2536 scope.go:117] "RemoveContainer" containerID="b1f17a9836c6775eaec75e88134b868b9ffc04cff95c9f81903c2635677896d0" Sep 16 17:28:37.430900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:37.430863 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fsg5h" event={"ID":"4d026f29-dab6-48ed-b64c-83dc0670f229","Type":"ContainerStarted","Data":"3b925429bb484f87db162028a53592060a741fefd6084f37663a99b5420552b5"} Sep 16 17:28:39.182051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:39.182020 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:28:39.185725 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:28:39.185704 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6858ddbdb-cxvw7" Sep 16 17:29:00.490306 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.490270 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679b59c599-x4r4h" podUID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" containerName="console" containerID="cri-o://9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a" gracePeriod=15 Sep 16 17:29:00.762332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.762265 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:29:00.904474 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904436 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.904659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904505 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.904659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904555 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.904659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904615 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7wwp\" (UniqueName: \"kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.904659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904643 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.904861 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.904670 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config\") pod \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\" (UID: \"2f5221f7-ea01-437d-bcb8-a575165c1ffa\") " Sep 16 17:29:00.905085 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.905064 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:00.905183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.905061 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:00.905183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.905113 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config" (OuterVolumeSpecName: "console-config") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:00.906778 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.906754 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:29:00.907065 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.907040 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:29:00.907165 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:00.907124 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp" (OuterVolumeSpecName: "kube-api-access-d7wwp") pod "2f5221f7-ea01-437d-bcb8-a575165c1ffa" (UID: "2f5221f7-ea01-437d-bcb8-a575165c1ffa"). InnerVolumeSpecName "kube-api-access-d7wwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:29:01.005790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005756 2536 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.005790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005784 2536 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-oauth-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.005790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005793 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wwp\" (UniqueName: \"kubernetes.io/projected/2f5221f7-ea01-437d-bcb8-a575165c1ffa-kube-api-access-d7wwp\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.006093 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005801 2536 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-service-ca\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.006093 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005810 2536 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.006093 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.005818 2536 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f5221f7-ea01-437d-bcb8-a575165c1ffa-console-oauth-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:01.523019 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.522989 2536 generic.go:356] "Generic (PLEG): container finished" podID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" containerID="9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a" exitCode=0 Sep 16 17:29:01.523408 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.523034 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b59c599-x4r4h" Sep 16 17:29:01.523408 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.523038 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b59c599-x4r4h" event={"ID":"2f5221f7-ea01-437d-bcb8-a575165c1ffa","Type":"ContainerDied","Data":"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a"} Sep 16 17:29:01.523408 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.523135 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b59c599-x4r4h" event={"ID":"2f5221f7-ea01-437d-bcb8-a575165c1ffa","Type":"ContainerDied","Data":"3e61c8028810bddef0112fcce5798b26baaf1f4a385c68046a16db8db47fef69"} Sep 16 17:29:01.523408 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.523153 2536 scope.go:117] "RemoveContainer" containerID="9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a" Sep 16 17:29:01.533205 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.533187 2536 scope.go:117] "RemoveContainer" containerID="9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a" Sep 16 17:29:01.533465 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:29:01.533447 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a\": container with ID starting with 9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a not found: ID does not exist" containerID="9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a" Sep 16 17:29:01.533513 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.533483 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a"} err="failed to get container status \"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a\": rpc error: code = NotFound desc = could not find container \"9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a\": container with ID starting with 9675e80794c2b782332adb331483c4b0f4c7dda941d2abbefc71286b8f36279a not found: ID does not exist" Sep 16 17:29:01.547895 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.547867 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:29:01.551110 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:01.551087 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679b59c599-x4r4h"] Sep 16 17:29:02.476485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:02.476448 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" path="/var/lib/kubelet/pods/2f5221f7-ea01-437d-bcb8-a575165c1ffa/volumes" Sep 16 17:29:03.322817 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:03.322779 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:29:03.325020 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:03.324997 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cae5e35-ada9-445e-b9a6-7876dee2d9ff-metrics-certs\") pod \"network-metrics-daemon-vxj45\" (UID: \"4cae5e35-ada9-445e-b9a6-7876dee2d9ff\") " pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:29:03.475166 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:03.475138 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhb2d\"" Sep 16 17:29:03.483440 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:03.483423 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxj45" Sep 16 17:29:03.784288 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:03.784252 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxj45"] Sep 16 17:29:03.788452 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:29:03.788423 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cae5e35_ada9_445e_b9a6_7876dee2d9ff.slice/crio-fbd9bde2652e3d944758474fb04d6b776cfbf4831384f98e336fe6b308a65518 WatchSource:0}: Error finding container fbd9bde2652e3d944758474fb04d6b776cfbf4831384f98e336fe6b308a65518: Status 404 returned error can't find the container with id fbd9bde2652e3d944758474fb04d6b776cfbf4831384f98e336fe6b308a65518 Sep 16 17:29:04.535840 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:04.535807 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxj45" event={"ID":"4cae5e35-ada9-445e-b9a6-7876dee2d9ff","Type":"ContainerStarted","Data":"fbd9bde2652e3d944758474fb04d6b776cfbf4831384f98e336fe6b308a65518"} Sep 16 17:29:05.541050 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:05.541012 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxj45" event={"ID":"4cae5e35-ada9-445e-b9a6-7876dee2d9ff","Type":"ContainerStarted","Data":"f01cdd156c3913f6096b3a0cabc47f0bda826e26cf2e2250c998a4b3499013e0"} Sep 16 17:29:05.541493 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:05.541063 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxj45" event={"ID":"4cae5e35-ada9-445e-b9a6-7876dee2d9ff","Type":"ContainerStarted","Data":"148c74ab5073ffb670990db7c88dbf2e2e081bd254fd21eff028166ed22820e3"} Sep 16 17:29:05.557098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:05.557044 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vxj45" podStartSLOduration=251.193844276 podStartE2EDuration="4m12.557028533s" totalImagesPullingTime="1.363184257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:05.556534434 +0000 UTC m=+253.499856063" watchObservedRunningTime="2026-09-16 17:29:05.557028533 +0000 UTC m=+253.500350135" Sep 16 17:29:21.453270 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.453195 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:29:21.453694 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.453451 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" containerName="console" Sep 16 17:29:21.453694 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.453462 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" containerName="console" Sep 16 17:29:21.453694 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.453506 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5221f7-ea01-437d-bcb8-a575165c1ffa" containerName="console" Sep 16 17:29:21.455230 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.455214 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.467718 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.467689 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:29:21.557294 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557254 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptbv\" (UniqueName: \"kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557462 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557304 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557462 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557346 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557462 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557413 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557462 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557450 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557607 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557466 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.557607 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.557510 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658044 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658083 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658100 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658341 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658297 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptbv\" (UniqueName: \"kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658375 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658367 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658422 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658408 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658461 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658448 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.658890 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.658865 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.659103 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.659087 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.659287 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.659262 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.659327 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.659267 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.660774 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.660753 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.660854 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.660804 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.670540 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.670512 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptbv\" (UniqueName: \"kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv\") pod \"console-6447565549-kbv52\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:21.763739 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:21.763701 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:22.034175 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:22.034105 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:29:22.037359 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:29:22.037333 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fb9786_a47d_424d_b48d_16a2149a6202.slice/crio-24f4da16ab6dce4950a97dd20ffd477bf84926d17789a73e7924f55bcac3ef8c WatchSource:0}: Error finding container 24f4da16ab6dce4950a97dd20ffd477bf84926d17789a73e7924f55bcac3ef8c: Status 404 returned error can't find the container with id 24f4da16ab6dce4950a97dd20ffd477bf84926d17789a73e7924f55bcac3ef8c Sep 16 17:29:22.601359 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:22.601323 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447565549-kbv52" event={"ID":"98fb9786-a47d-424d-b48d-16a2149a6202","Type":"ContainerStarted","Data":"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9"} Sep 16 17:29:22.601714 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:22.601384 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447565549-kbv52" event={"ID":"98fb9786-a47d-424d-b48d-16a2149a6202","Type":"ContainerStarted","Data":"24f4da16ab6dce4950a97dd20ffd477bf84926d17789a73e7924f55bcac3ef8c"} Sep 16 17:29:22.622719 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:22.622661 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6447565549-kbv52" podStartSLOduration=1.622642469 podStartE2EDuration="1.622642469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:22.622010393 +0000 UTC m=+270.565332012" watchObservedRunningTime="2026-09-16 17:29:22.622642469 +0000 UTC m=+270.565964072" Sep 16 17:29:31.764297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:31.764257 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:31.764682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:31.764351 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:31.769747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:31.769723 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:32.640559 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:32.640532 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:29:32.682207 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:32.682177 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:29:52.397752 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:52.397726 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:29:52.400743 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:52.400718 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:29:52.401171 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:52.401147 2536 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 17:29:57.704166 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:57.704120 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-646b9ff67-zzs7r" podUID="41be4528-44f6-4221-be83-dbc721b2384b" containerName="console" containerID="cri-o://eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f" gracePeriod=15 Sep 16 17:29:57.936455 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:57.936430 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:29:58.045368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045329 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045393 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045410 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045447 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045489 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9w2l\" (UniqueName: \"kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045528 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.045591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045549 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert\") pod \"41be4528-44f6-4221-be83-dbc721b2384b\" (UID: \"41be4528-44f6-4221-be83-dbc721b2384b\") " Sep 16 17:29:58.046020 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.045977 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:58.046126 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.046031 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca" (OuterVolumeSpecName: "service-ca") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:58.046126 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.046043 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config" (OuterVolumeSpecName: "console-config") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:58.046126 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.046078 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:29:58.047767 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.047730 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l" (OuterVolumeSpecName: "kube-api-access-b9w2l") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "kube-api-access-b9w2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:29:58.047853 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.047761 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:29:58.047853 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.047806 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41be4528-44f6-4221-be83-dbc721b2384b" (UID: "41be4528-44f6-4221-be83-dbc721b2384b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:29:58.146599 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146561 2536 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-console-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146599 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146594 2536 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-oauth-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146599 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146604 2536 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-service-ca\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146613 2536 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146623 2536 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41be4528-44f6-4221-be83-dbc721b2384b-trusted-ca-bundle\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146631 2536 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41be4528-44f6-4221-be83-dbc721b2384b-console-oauth-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.146797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.146640 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9w2l\" (UniqueName: \"kubernetes.io/projected/41be4528-44f6-4221-be83-dbc721b2384b-kube-api-access-b9w2l\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:29:58.729396 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.729360 2536 generic.go:356] "Generic (PLEG): container finished" podID="41be4528-44f6-4221-be83-dbc721b2384b" containerID="eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f" exitCode=0 Sep 16 17:29:58.729872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.729402 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646b9ff67-zzs7r" Sep 16 17:29:58.729872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.729441 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646b9ff67-zzs7r" event={"ID":"41be4528-44f6-4221-be83-dbc721b2384b","Type":"ContainerDied","Data":"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f"} Sep 16 17:29:58.729872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.729495 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646b9ff67-zzs7r" event={"ID":"41be4528-44f6-4221-be83-dbc721b2384b","Type":"ContainerDied","Data":"828a85b8d654021a404eb11751a24ace76afff8794c44bf6e5cce9a9bb945a31"} Sep 16 17:29:58.729872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.729517 2536 scope.go:117] "RemoveContainer" containerID="eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f" Sep 16 17:29:58.737486 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.737457 2536 scope.go:117] "RemoveContainer" containerID="eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f" Sep 16 17:29:58.737716 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:29:58.737695 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f\": container with ID starting with eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f not found: ID does not exist" containerID="eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f" Sep 16 17:29:58.737778 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.737727 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f"} err="failed to get container status \"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f\": rpc error: code = NotFound desc = could not find container \"eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f\": container with ID starting with eaa8f93891a15f7e08cb830c6c51ce71dc1a4653886130fd052604dc7254952f not found: ID does not exist" Sep 16 17:29:58.748234 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.748212 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:29:58.752052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:29:58.752031 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-646b9ff67-zzs7r"] Sep 16 17:30:00.477111 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:00.477078 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41be4528-44f6-4221-be83-dbc721b2384b" path="/var/lib/kubelet/pods/41be4528-44f6-4221-be83-dbc721b2384b/volumes" Sep 16 17:30:38.944155 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.944117 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:30:38.944705 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.944484 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41be4528-44f6-4221-be83-dbc721b2384b" containerName="console" Sep 16 17:30:38.944705 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.944502 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="41be4528-44f6-4221-be83-dbc721b2384b" containerName="console" Sep 16 17:30:38.944705 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.944567 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="41be4528-44f6-4221-be83-dbc721b2384b" containerName="console" Sep 16 17:30:38.946377 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.946354 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:38.968664 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:38.968639 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:30:39.047197 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047164 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hhq\" (UniqueName: \"kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047212 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047270 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047308 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047346 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047377 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.047497 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.047405 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148239 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148197 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148255 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148278 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148308 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hhq\" (UniqueName: \"kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148335 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148354 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.148403 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.148383 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.149187 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.149165 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.149187 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.149180 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.149316 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.149211 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.149354 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.149331 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.150881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.150848 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.151008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.150962 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.159899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.159879 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hhq\" (UniqueName: \"kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq\") pod \"console-b6b95d58f-khg85\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.255511 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.255472 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:39.521532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.521454 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:30:39.524531 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:30:39.524505 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce201115_8db1_4dde_b413_199ae2c9e693.slice/crio-7c420135d787b799999ab025fc08f0efae62b1fbc73cb93eb6a81c9a52c3e479 WatchSource:0}: Error finding container 7c420135d787b799999ab025fc08f0efae62b1fbc73cb93eb6a81c9a52c3e479: Status 404 returned error can't find the container with id 7c420135d787b799999ab025fc08f0efae62b1fbc73cb93eb6a81c9a52c3e479 Sep 16 17:30:39.869995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.869881 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6b95d58f-khg85" event={"ID":"ce201115-8db1-4dde-b413-199ae2c9e693","Type":"ContainerStarted","Data":"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11"} Sep 16 17:30:39.869995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.869954 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6b95d58f-khg85" event={"ID":"ce201115-8db1-4dde-b413-199ae2c9e693","Type":"ContainerStarted","Data":"7c420135d787b799999ab025fc08f0efae62b1fbc73cb93eb6a81c9a52c3e479"} Sep 16 17:30:39.888153 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:39.888108 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b6b95d58f-khg85" podStartSLOduration=1.888093655 podStartE2EDuration="1.888093655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:30:39.886875258 +0000 UTC m=+347.830196872" watchObservedRunningTime="2026-09-16 17:30:39.888093655 +0000 UTC m=+347.831415258" Sep 16 17:30:49.256576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:49.256532 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:49.257083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:49.256605 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:49.261274 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:49.261252 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:49.909671 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:49.909644 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:30:49.966600 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:30:49.966566 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:31:14.986531 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:14.986470 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6447565549-kbv52" podUID="98fb9786-a47d-424d-b48d-16a2149a6202" containerName="console" containerID="cri-o://8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9" gracePeriod=15 Sep 16 17:31:15.219029 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.219004 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:31:15.356200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356110 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356161 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356200 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356477 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356220 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rptbv\" (UniqueName: \"kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356477 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356266 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356477 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356280 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356477 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356309 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca\") pod \"98fb9786-a47d-424d-b48d-16a2149a6202\" (UID: \"98fb9786-a47d-424d-b48d-16a2149a6202\") " Sep 16 17:31:15.356738 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356699 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config" (OuterVolumeSpecName: "console-config") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:31:15.356738 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356734 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca" (OuterVolumeSpecName: "service-ca") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:31:15.357019 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356706 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:31:15.357019 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.356716 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:31:15.358359 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.358336 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:31:15.358456 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.358426 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv" (OuterVolumeSpecName: "kube-api-access-rptbv") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "kube-api-access-rptbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:31:15.358501 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.358471 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98fb9786-a47d-424d-b48d-16a2149a6202" (UID: "98fb9786-a47d-424d-b48d-16a2149a6202"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:31:15.457214 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457168 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptbv\" (UniqueName: \"kubernetes.io/projected/98fb9786-a47d-424d-b48d-16a2149a6202-kube-api-access-rptbv\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457214 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457210 2536 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-trusted-ca-bundle\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457214 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457220 2536 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457214 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457230 2536 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-service-ca\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457466 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457238 2536 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98fb9786-a47d-424d-b48d-16a2149a6202-console-oauth-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457466 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457247 2536 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-console-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:15.457466 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:15.457256 2536 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98fb9786-a47d-424d-b48d-16a2149a6202-oauth-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:31:16.005667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.005633 2536 generic.go:356] "Generic (PLEG): container finished" podID="98fb9786-a47d-424d-b48d-16a2149a6202" containerID="8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9" exitCode=0 Sep 16 17:31:16.006068 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.005676 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447565549-kbv52" Sep 16 17:31:16.006068 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.005722 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447565549-kbv52" event={"ID":"98fb9786-a47d-424d-b48d-16a2149a6202","Type":"ContainerDied","Data":"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9"} Sep 16 17:31:16.006068 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.005774 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447565549-kbv52" event={"ID":"98fb9786-a47d-424d-b48d-16a2149a6202","Type":"ContainerDied","Data":"24f4da16ab6dce4950a97dd20ffd477bf84926d17789a73e7924f55bcac3ef8c"} Sep 16 17:31:16.006068 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.005799 2536 scope.go:117] "RemoveContainer" containerID="8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9" Sep 16 17:31:16.013845 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.013827 2536 scope.go:117] "RemoveContainer" containerID="8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9" Sep 16 17:31:16.014135 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:31:16.014119 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9\": container with ID starting with 8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9 not found: ID does not exist" containerID="8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9" Sep 16 17:31:16.014175 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.014141 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9"} err="failed to get container status \"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9\": rpc error: code = NotFound desc = could not find container \"8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9\": container with ID starting with 8ecea8d4a6c699f16affbb3f9fd46bf97800fc98a0983c06f5f0bea0cad610a9 not found: ID does not exist" Sep 16 17:31:16.025593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.025570 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:31:16.032943 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.031770 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6447565549-kbv52"] Sep 16 17:31:16.476456 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:16.476427 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fb9786-a47d-424d-b48d-16a2149a6202" path="/var/lib/kubelet/pods/98fb9786-a47d-424d-b48d-16a2149a6202/volumes" Sep 16 17:31:56.285075 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.285042 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj"] Sep 16 17:31:56.285450 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.285314 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fb9786-a47d-424d-b48d-16a2149a6202" containerName="console" Sep 16 17:31:56.285450 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.285325 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fb9786-a47d-424d-b48d-16a2149a6202" containerName="console" Sep 16 17:31:56.285450 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.285370 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fb9786-a47d-424d-b48d-16a2149a6202" containerName="console" Sep 16 17:31:56.289612 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.289595 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.292568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.292545 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 17:31:56.292688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.292614 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bp5jk\"" Sep 16 17:31:56.292688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.292653 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 17:31:56.300030 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.300009 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj"] Sep 16 17:31:56.348638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.348603 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.348638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.348640 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.348832 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.348669 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xgz\" (UniqueName: \"kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.449941 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.449881 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.450099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.449948 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.450099 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.449995 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xgz\" (UniqueName: \"kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.450286 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.450267 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.450355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.450334 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.459055 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.459035 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xgz\" (UniqueName: \"kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.598492 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.598398 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:31:56.868541 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.868383 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj"] Sep 16 17:31:56.871171 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:31:56.871144 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4926b54b_fd2a_438b_949a_208e61708795.slice/crio-9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca WatchSource:0}: Error finding container 9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca: Status 404 returned error can't find the container with id 9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca Sep 16 17:31:56.873033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:56.873013 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 17:31:57.156580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:31:57.156503 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" event={"ID":"4926b54b-fd2a-438b-949a-208e61708795","Type":"ContainerStarted","Data":"9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca"} Sep 16 17:32:03.180699 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:03.180661 2536 generic.go:356] "Generic (PLEG): container finished" podID="4926b54b-fd2a-438b-949a-208e61708795" containerID="94bf6699d4aed05c81e2842ad0d5fea60bd6f53c0432572ca915ca8736677c26" exitCode=0 Sep 16 17:32:03.181116 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:03.180721 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" event={"ID":"4926b54b-fd2a-438b-949a-208e61708795","Type":"ContainerDied","Data":"94bf6699d4aed05c81e2842ad0d5fea60bd6f53c0432572ca915ca8736677c26"} Sep 16 17:32:06.195429 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:06.195392 2536 generic.go:356] "Generic (PLEG): container finished" podID="4926b54b-fd2a-438b-949a-208e61708795" containerID="d8497acbc440cb8497388416ceeac47417cdf8f0132cc091924cf03d35c29950" exitCode=0 Sep 16 17:32:06.195802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:06.195451 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" event={"ID":"4926b54b-fd2a-438b-949a-208e61708795","Type":"ContainerDied","Data":"d8497acbc440cb8497388416ceeac47417cdf8f0132cc091924cf03d35c29950"} Sep 16 17:32:13.224728 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:13.224691 2536 generic.go:356] "Generic (PLEG): container finished" podID="4926b54b-fd2a-438b-949a-208e61708795" containerID="1b37f345d845210169381f85ec203bc4222fcd1ab7a66e2aa84d84151e2d4905" exitCode=0 Sep 16 17:32:13.225097 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:13.224741 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" event={"ID":"4926b54b-fd2a-438b-949a-208e61708795","Type":"ContainerDied","Data":"1b37f345d845210169381f85ec203bc4222fcd1ab7a66e2aa84d84151e2d4905"} Sep 16 17:32:14.352788 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.352764 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:32:14.510640 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.510554 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util\") pod \"4926b54b-fd2a-438b-949a-208e61708795\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " Sep 16 17:32:14.510640 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.510609 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66xgz\" (UniqueName: \"kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz\") pod \"4926b54b-fd2a-438b-949a-208e61708795\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " Sep 16 17:32:14.510834 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.510650 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle\") pod \"4926b54b-fd2a-438b-949a-208e61708795\" (UID: \"4926b54b-fd2a-438b-949a-208e61708795\") " Sep 16 17:32:14.511345 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.511317 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle" (OuterVolumeSpecName: "bundle") pod "4926b54b-fd2a-438b-949a-208e61708795" (UID: "4926b54b-fd2a-438b-949a-208e61708795"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:32:14.512818 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.512792 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz" (OuterVolumeSpecName: "kube-api-access-66xgz") pod "4926b54b-fd2a-438b-949a-208e61708795" (UID: "4926b54b-fd2a-438b-949a-208e61708795"). InnerVolumeSpecName "kube-api-access-66xgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:32:14.514579 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.514559 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util" (OuterVolumeSpecName: "util") pod "4926b54b-fd2a-438b-949a-208e61708795" (UID: "4926b54b-fd2a-438b-949a-208e61708795"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:32:14.611500 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.611462 2536 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-util\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:32:14.611500 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.611500 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xgz\" (UniqueName: \"kubernetes.io/projected/4926b54b-fd2a-438b-949a-208e61708795-kube-api-access-66xgz\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:32:14.611731 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:14.611515 2536 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4926b54b-fd2a-438b-949a-208e61708795-bundle\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:32:15.233702 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:15.233667 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" event={"ID":"4926b54b-fd2a-438b-949a-208e61708795","Type":"ContainerDied","Data":"9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca"} Sep 16 17:32:15.233702 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:15.233697 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a7jjnj" Sep 16 17:32:15.233908 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:15.233714 2536 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe7738470fe1bb33667f7b1b54430285738bde38e01e245970ca214bcf2b9ca" Sep 16 17:32:19.807343 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807308 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt"] Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807574 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="extract" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807586 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="extract" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807596 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="util" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807601 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="util" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807619 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="pull" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807624 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="pull" Sep 16 17:32:19.807703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.807664 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4926b54b-fd2a-438b-949a-208e61708795" containerName="extract" Sep 16 17:32:19.858866 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.858831 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt"] Sep 16 17:32:19.859034 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.858949 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:19.863008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.862985 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 17:32:19.863129 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.863011 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 17:32:19.863597 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.863578 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-7mp9r\"" Sep 16 17:32:19.864161 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.864138 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 17:32:19.959848 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.959812 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:19.960034 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:19.959872 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhw6\" (UniqueName: \"kubernetes.io/projected/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-kube-api-access-vwhw6\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.061216 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.061134 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhw6\" (UniqueName: \"kubernetes.io/projected/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-kube-api-access-vwhw6\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.061216 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.061188 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.063425 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.063401 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.086820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.086792 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhw6\" (UniqueName: \"kubernetes.io/projected/c7f2c898-9592-4cf9-9e1e-791ee8646ba9-kube-api-access-vwhw6\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-wnrbt\" (UID: \"c7f2c898-9592-4cf9-9e1e-791ee8646ba9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.168541 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.168503 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:20.480805 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:20.480773 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt"] Sep 16 17:32:20.484621 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:32:20.484597 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f2c898_9592_4cf9_9e1e_791ee8646ba9.slice/crio-467f4308d8ac0ed9ec1008fd008b4f73648f5eb8ea87085edb9a3a4f6660b5be WatchSource:0}: Error finding container 467f4308d8ac0ed9ec1008fd008b4f73648f5eb8ea87085edb9a3a4f6660b5be: Status 404 returned error can't find the container with id 467f4308d8ac0ed9ec1008fd008b4f73648f5eb8ea87085edb9a3a4f6660b5be Sep 16 17:32:21.256541 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:21.256508 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" event={"ID":"c7f2c898-9592-4cf9-9e1e-791ee8646ba9","Type":"ContainerStarted","Data":"467f4308d8ac0ed9ec1008fd008b4f73648f5eb8ea87085edb9a3a4f6660b5be"} Sep 16 17:32:25.269948 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.269895 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d2g27"] Sep 16 17:32:25.291786 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.291747 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d2g27"] Sep 16 17:32:25.291934 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.291794 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" event={"ID":"c7f2c898-9592-4cf9-9e1e-791ee8646ba9","Type":"ContainerStarted","Data":"5c3fa22205edc5360a98cf520841d51545f73be95f7d2d34019e45a151eada51"} Sep 16 17:32:25.291934 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.291834 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:25.291934 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.291903 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.298705 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.298682 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 17:32:25.298850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.298720 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6spxq\"" Sep 16 17:32:25.298850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.298812 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 16 17:32:25.311333 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.311285 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" podStartSLOduration=2.469829875 podStartE2EDuration="6.311268544s" totalImagesPullingTime="3.841438669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:32:25.309813555 +0000 UTC m=+453.253135174" watchObservedRunningTime="2026-09-16 17:32:25.311268544 +0000 UTC m=+453.254590147" Sep 16 17:32:25.404976 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.404940 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eee0f005-94b7-4e69-9d69-a847940dd9e2-cabundle0\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.405165 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.405031 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-kube-api-access-9vvwq\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.405165 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.405063 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.506113 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.506070 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-kube-api-access-9vvwq\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.506297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.506121 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.506297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.506215 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eee0f005-94b7-4e69-9d69-a847940dd9e2-cabundle0\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.506297 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:25.506265 2536 secret.go:281] references non-existent secret key: ca.crt Sep 16 17:32:25.506297 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:25.506289 2536 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 17:32:25.506502 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:25.506301 2536 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-d2g27: references non-existent secret key: ca.crt Sep 16 17:32:25.506502 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:25.506377 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates podName:eee0f005-94b7-4e69-9d69-a847940dd9e2 nodeName:}" failed. No retries permitted until 2026-09-16 17:32:26.006356026 +0000 UTC m=+453.949677622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates") pod "keda-operator-579c8958f7-d2g27" (UID: "eee0f005-94b7-4e69-9d69-a847940dd9e2") : references non-existent secret key: ca.crt Sep 16 17:32:25.506810 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.506792 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eee0f005-94b7-4e69-9d69-a847940dd9e2-cabundle0\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:25.521546 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:25.521490 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-kube-api-access-9vvwq\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:26.010894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.010862 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:26.013231 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.013210 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eee0f005-94b7-4e69-9d69-a847940dd9e2-certificates\") pod \"keda-operator-579c8958f7-d2g27\" (UID: \"eee0f005-94b7-4e69-9d69-a847940dd9e2\") " pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:26.144982 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.144946 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pvwjp"] Sep 16 17:32:26.165512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.165484 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pvwjp"] Sep 16 17:32:26.165679 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.165586 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.168642 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.168621 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 16 17:32:26.204051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.204024 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:26.313929 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.313894 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zd22\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-kube-api-access-6zd22\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.314276 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.313943 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.414449 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.414404 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zd22\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-kube-api-access-6zd22\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.414622 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.414457 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.414692 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.414625 2536 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Sep 16 17:32:26.414692 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.414651 2536 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-pvwjp: secret "keda-admission-webhooks-certs" not found Sep 16 17:32:26.414803 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.414715 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates podName:6ce035a1-005f-491f-839a-ef644222d860 nodeName:}" failed. No retries permitted until 2026-09-16 17:32:26.914692562 +0000 UTC m=+454.858014158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates") pod "keda-admission-6c4d864b86-pvwjp" (UID: "6ce035a1-005f-491f-839a-ef644222d860") : secret "keda-admission-webhooks-certs" not found Sep 16 17:32:26.426069 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.426038 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zd22\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-kube-api-access-6zd22\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.493182 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.493151 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d2g27"] Sep 16 17:32:26.496287 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:32:26.496259 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee0f005_94b7_4e69_9d69_a847940dd9e2.slice/crio-c35ca2ca0fecb9aac1d8d60bf64e09a8e8d2ff26fdf6ad65d0ddc349b66d6d55 WatchSource:0}: Error finding container c35ca2ca0fecb9aac1d8d60bf64e09a8e8d2ff26fdf6ad65d0ddc349b66d6d55: Status 404 returned error can't find the container with id c35ca2ca0fecb9aac1d8d60bf64e09a8e8d2ff26fdf6ad65d0ddc349b66d6d55 Sep 16 17:32:26.919012 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:26.918974 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:26.919183 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.919107 2536 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Sep 16 17:32:26.919183 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.919127 2536 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-pvwjp: secret "keda-admission-webhooks-certs" not found Sep 16 17:32:26.919254 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:32:26.919188 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates podName:6ce035a1-005f-491f-839a-ef644222d860 nodeName:}" failed. No retries permitted until 2026-09-16 17:32:27.919172112 +0000 UTC m=+455.862493694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates") pod "keda-admission-6c4d864b86-pvwjp" (UID: "6ce035a1-005f-491f-839a-ef644222d860") : secret "keda-admission-webhooks-certs" not found Sep 16 17:32:27.287765 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:27.287732 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d2g27" event={"ID":"eee0f005-94b7-4e69-9d69-a847940dd9e2","Type":"ContainerStarted","Data":"c35ca2ca0fecb9aac1d8d60bf64e09a8e8d2ff26fdf6ad65d0ddc349b66d6d55"} Sep 16 17:32:27.927091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:27.927048 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:27.929852 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:27.929824 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6ce035a1-005f-491f-839a-ef644222d860-certificates\") pod \"keda-admission-6c4d864b86-pvwjp\" (UID: \"6ce035a1-005f-491f-839a-ef644222d860\") " pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:27.975771 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:27.975726 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:28.266990 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:28.266957 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pvwjp"] Sep 16 17:32:28.270528 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:32:28.270495 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce035a1_005f_491f_839a_ef644222d860.slice/crio-1e53441c10b1027d281292846ebad33e4dbb2856dcb946bd2e07df6761746133 WatchSource:0}: Error finding container 1e53441c10b1027d281292846ebad33e4dbb2856dcb946bd2e07df6761746133: Status 404 returned error can't find the container with id 1e53441c10b1027d281292846ebad33e4dbb2856dcb946bd2e07df6761746133 Sep 16 17:32:28.293988 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:28.293948 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" event={"ID":"6ce035a1-005f-491f-839a-ef644222d860","Type":"ContainerStarted","Data":"1e53441c10b1027d281292846ebad33e4dbb2856dcb946bd2e07df6761746133"} Sep 16 17:32:30.304098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:30.304060 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d2g27" event={"ID":"eee0f005-94b7-4e69-9d69-a847940dd9e2","Type":"ContainerStarted","Data":"8f5a203f57de6ec116726f99373c184fb1fd637efc5b0235e08634827a140dc2"} Sep 16 17:32:30.304479 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:30.304185 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:30.326046 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:30.325994 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-d2g27" podStartSLOduration=2.065017303 podStartE2EDuration="5.325978474s" totalImagesPullingTime="3.260961171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:32:30.324872611 +0000 UTC m=+458.268194236" watchObservedRunningTime="2026-09-16 17:32:30.325978474 +0000 UTC m=+458.269300073" Sep 16 17:32:37.339432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:37.339387 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" event={"ID":"6ce035a1-005f-491f-839a-ef644222d860","Type":"ContainerStarted","Data":"06fb496a9d8ab0b40aff5027857c4960130d26975eb1dac74e89bf908e61d0b2"} Sep 16 17:32:37.339862 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:37.339512 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:32:37.360970 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:37.360887 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" podStartSLOduration=2.774158686 podStartE2EDuration="11.360865995s" totalImagesPullingTime="8.586707309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:32:37.358739432 +0000 UTC m=+465.302061063" watchObservedRunningTime="2026-09-16 17:32:37.360865995 +0000 UTC m=+465.304187598" Sep 16 17:32:46.285547 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:46.285511 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-wnrbt" Sep 16 17:32:51.310812 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:51.310777 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-d2g27" Sep 16 17:32:58.346835 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:32:58.346802 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-pvwjp" Sep 16 17:33:21.426507 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.426476 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7df64f45-jzlmd"] Sep 16 17:33:21.428461 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.428443 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.444277 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.444249 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7df64f45-jzlmd"] Sep 16 17:33:21.568415 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568381 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528q2\" (UniqueName: \"kubernetes.io/projected/8863c674-a354-4b62-8fab-0cb963cccab3-kube-api-access-528q2\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568425 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568446 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-trusted-ca-bundle\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568525 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-oauth-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568692 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568592 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-oauth-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568692 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568628 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-console-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.568692 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.568644 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-service-ca\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.669741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669698 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.669741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669744 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-trusted-ca-bundle\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670007 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669769 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-oauth-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670007 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669812 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-oauth-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670007 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669854 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-console-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670007 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.669875 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-service-ca\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670177 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.670015 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528q2\" (UniqueName: \"kubernetes.io/projected/8863c674-a354-4b62-8fab-0cb963cccab3-kube-api-access-528q2\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.670645 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-console-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.670655 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-service-ca\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670788 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.670649 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-oauth-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.670788 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.670712 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8863c674-a354-4b62-8fab-0cb963cccab3-trusted-ca-bundle\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.672640 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.672616 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-oauth-config\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.672724 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.672696 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8863c674-a354-4b62-8fab-0cb963cccab3-console-serving-cert\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.685400 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.685350 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528q2\" (UniqueName: \"kubernetes.io/projected/8863c674-a354-4b62-8fab-0cb963cccab3-kube-api-access-528q2\") pod \"console-7b7df64f45-jzlmd\" (UID: \"8863c674-a354-4b62-8fab-0cb963cccab3\") " pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:21.737153 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:21.737118 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:22.041894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:22.041861 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7df64f45-jzlmd"] Sep 16 17:33:22.047335 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:33:22.047306 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8863c674_a354_4b62_8fab_0cb963cccab3.slice/crio-477ff58c0e3192ccdfb958a9d97e2c2a2b353ea39e276b6febcf00fc831ed627 WatchSource:0}: Error finding container 477ff58c0e3192ccdfb958a9d97e2c2a2b353ea39e276b6febcf00fc831ed627: Status 404 returned error can't find the container with id 477ff58c0e3192ccdfb958a9d97e2c2a2b353ea39e276b6febcf00fc831ed627 Sep 16 17:33:22.516703 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:22.516667 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7df64f45-jzlmd" event={"ID":"8863c674-a354-4b62-8fab-0cb963cccab3","Type":"ContainerStarted","Data":"57fa23e01da51509282af1eb46038e52bbe0672fe5149ea912a5b4982b944d0a"} Sep 16 17:33:22.517197 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:22.516714 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7df64f45-jzlmd" event={"ID":"8863c674-a354-4b62-8fab-0cb963cccab3","Type":"ContainerStarted","Data":"477ff58c0e3192ccdfb958a9d97e2c2a2b353ea39e276b6febcf00fc831ed627"} Sep 16 17:33:31.737351 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:31.737312 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:31.737846 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:31.737460 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:31.741974 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:31.741946 2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:31.836579 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:31.836522 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7df64f45-jzlmd" podStartSLOduration=10.83650356 podStartE2EDuration="10.83650356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:22.596794608 +0000 UTC m=+510.540116230" watchObservedRunningTime="2026-09-16 17:33:31.83650356 +0000 UTC m=+519.779825162" Sep 16 17:33:32.561567 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:32.561537 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7df64f45-jzlmd" Sep 16 17:33:32.675949 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:32.675895 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:33:51.008013 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.007975 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:33:51.009951 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.009936 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.014232 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.014211 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 17:33:51.014384 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.014269 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-nmwtk\"" Sep 16 17:33:51.014432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.014394 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 17:33:51.014432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.014398 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 17:33:51.029344 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.029312 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:33:51.057025 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.056996 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-w5wv9"] Sep 16 17:33:51.059027 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.059011 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.061544 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.061524 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 17:33:51.061797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.061785 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\"" Sep 16 17:33:51.075539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.075516 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w5wv9"] Sep 16 17:33:51.106140 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.106109 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fee5c41-9fa1-4e79-bb11-05fba6118799-data\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.106281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.106164 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldld\" (UniqueName: \"kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.106281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.106186 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.106281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.106231 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jv8j\" (UniqueName: \"kubernetes.io/projected/4fee5c41-9fa1-4e79-bb11-05fba6118799-kube-api-access-5jv8j\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.207628 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.207581 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldld\" (UniqueName: \"kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.207628 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.207637 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.207901 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.207710 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jv8j\" (UniqueName: \"kubernetes.io/projected/4fee5c41-9fa1-4e79-bb11-05fba6118799-kube-api-access-5jv8j\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.207901 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.207743 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fee5c41-9fa1-4e79-bb11-05fba6118799-data\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.208170 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.208145 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fee5c41-9fa1-4e79-bb11-05fba6118799-data\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.210452 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.210430 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.217265 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.217238 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldld\" (UniqueName: \"kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld\") pod \"kserve-controller-manager-cc8594966-cqpxz\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.217365 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.217282 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jv8j\" (UniqueName: \"kubernetes.io/projected/4fee5c41-9fa1-4e79-bb11-05fba6118799-kube-api-access-5jv8j\") pod \"seaweedfs-64568bcd49-w5wv9\" (UID: \"4fee5c41-9fa1-4e79-bb11-05fba6118799\") " pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.319856 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.319774 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:51.368087 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.368060 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:51.589069 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.588988 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:33:51.592125 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:33:51.592096 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a09c12e_a243_48f1_a0ac_f3e9a200948f.slice/crio-21f3e59e5343ee8597327e7f7530fd877635f251c6f850a0f67f7193b7690c56 WatchSource:0}: Error finding container 21f3e59e5343ee8597327e7f7530fd877635f251c6f850a0f67f7193b7690c56: Status 404 returned error can't find the container with id 21f3e59e5343ee8597327e7f7530fd877635f251c6f850a0f67f7193b7690c56 Sep 16 17:33:51.645219 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.645176 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" event={"ID":"3a09c12e-a243-48f1-a0ac-f3e9a200948f","Type":"ContainerStarted","Data":"21f3e59e5343ee8597327e7f7530fd877635f251c6f850a0f67f7193b7690c56"} Sep 16 17:33:51.646290 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:51.646269 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w5wv9"] Sep 16 17:33:51.648405 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:33:51.648385 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fee5c41_9fa1_4e79_bb11_05fba6118799.slice/crio-b44735d7bcab4de1527972714121b592c4b053390ef8450d53a2b62334f19223 WatchSource:0}: Error finding container b44735d7bcab4de1527972714121b592c4b053390ef8450d53a2b62334f19223: Status 404 returned error can't find the container with id b44735d7bcab4de1527972714121b592c4b053390ef8450d53a2b62334f19223 Sep 16 17:33:52.653409 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:52.653366 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w5wv9" event={"ID":"4fee5c41-9fa1-4e79-bb11-05fba6118799","Type":"ContainerStarted","Data":"b44735d7bcab4de1527972714121b592c4b053390ef8450d53a2b62334f19223"} Sep 16 17:33:55.217198 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.217174 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 17:33:55.669899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.669797 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" event={"ID":"3a09c12e-a243-48f1-a0ac-f3e9a200948f","Type":"ContainerStarted","Data":"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909"} Sep 16 17:33:55.669899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.669867 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:33:55.671190 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.671169 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w5wv9" event={"ID":"4fee5c41-9fa1-4e79-bb11-05fba6118799","Type":"ContainerStarted","Data":"ce2c422685d6fe41c995ad2ae50ac6b2eddc9a7947b8623dc36757e9baff015b"} Sep 16 17:33:55.671292 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.671222 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:33:55.688251 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.688206 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" podStartSLOduration=2.149146468 podStartE2EDuration="5.688191386s" totalImagesPullingTime="3.539044918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:55.687083676 +0000 UTC m=+543.630405280" watchObservedRunningTime="2026-09-16 17:33:55.688191386 +0000 UTC m=+543.631512987" Sep 16 17:33:55.705083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:55.705039 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-w5wv9" podStartSLOduration=1.1402927 podStartE2EDuration="4.705024419s" totalImagesPullingTime="3.564731719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:55.704555898 +0000 UTC m=+543.647877530" watchObservedRunningTime="2026-09-16 17:33:55.705024419 +0000 UTC m=+543.648346020" Sep 16 17:33:57.694568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:57.694503 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b6b95d58f-khg85" podUID="ce201115-8db1-4dde-b413-199ae2c9e693" containerName="console" containerID="cri-o://22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11" gracePeriod=15 Sep 16 17:33:57.933637 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:57.933613 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:33:58.071094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071060 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071102 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071350 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071117 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071350 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071168 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071350 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071187 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5hhq\" (UniqueName: \"kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071350 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071206 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071350 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071255 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config\") pod \"ce201115-8db1-4dde-b413-199ae2c9e693\" (UID: \"ce201115-8db1-4dde-b413-199ae2c9e693\") " Sep 16 17:33:58.071676 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071642 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:33:58.071789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071736 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:33:58.071789 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071776 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:33:58.071888 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.071783 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config" (OuterVolumeSpecName: "console-config") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:33:58.073345 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.073318 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:33:58.073465 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.073379 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:33:58.073521 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.073467 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq" (OuterVolumeSpecName: "kube-api-access-q5hhq") pod "ce201115-8db1-4dde-b413-199ae2c9e693" (UID: "ce201115-8db1-4dde-b413-199ae2c9e693"). InnerVolumeSpecName "kube-api-access-q5hhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:33:58.172593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172562 2536 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-trusted-ca-bundle\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172588 2536 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172593 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172598 2536 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-console-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172608 2536 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-service-ca\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172616 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hhq\" (UniqueName: \"kubernetes.io/projected/ce201115-8db1-4dde-b413-199ae2c9e693-kube-api-access-q5hhq\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172624 2536 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce201115-8db1-4dde-b413-199ae2c9e693-oauth-serving-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.172806 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.172633 2536 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce201115-8db1-4dde-b413-199ae2c9e693-console-oauth-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:33:58.685628 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.685594 2536 generic.go:356] "Generic (PLEG): container finished" podID="ce201115-8db1-4dde-b413-199ae2c9e693" containerID="22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11" exitCode=0 Sep 16 17:33:58.685797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.685634 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6b95d58f-khg85" event={"ID":"ce201115-8db1-4dde-b413-199ae2c9e693","Type":"ContainerDied","Data":"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11"} Sep 16 17:33:58.685797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.685649 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6b95d58f-khg85" Sep 16 17:33:58.685797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.685672 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6b95d58f-khg85" event={"ID":"ce201115-8db1-4dde-b413-199ae2c9e693","Type":"ContainerDied","Data":"7c420135d787b799999ab025fc08f0efae62b1fbc73cb93eb6a81c9a52c3e479"} Sep 16 17:33:58.685797 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.685690 2536 scope.go:117] "RemoveContainer" containerID="22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11" Sep 16 17:33:58.693652 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.693635 2536 scope.go:117] "RemoveContainer" containerID="22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11" Sep 16 17:33:58.693934 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:33:58.693896 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11\": container with ID starting with 22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11 not found: ID does not exist" containerID="22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11" Sep 16 17:33:58.693980 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.693943 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11"} err="failed to get container status \"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11\": rpc error: code = NotFound desc = could not find container \"22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11\": container with ID starting with 22565c77d3a9596986d0372fcd3cbb533b95e8b9a02ef4c39135eccd2934ad11 not found: ID does not exist" Sep 16 17:33:58.703209 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.703187 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:33:58.707494 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:33:58.707474 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b6b95d58f-khg85"] Sep 16 17:34:00.476811 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:00.476778 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce201115-8db1-4dde-b413-199ae2c9e693" path="/var/lib/kubelet/pods/ce201115-8db1-4dde-b413-199ae2c9e693/volumes" Sep 16 17:34:01.678490 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:01.678462 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-w5wv9" Sep 16 17:34:26.681557 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:26.681526 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:34:38.861401 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.861368 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:34:38.861837 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.861548 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" podUID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" containerName="manager" containerID="cri-o://0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909" gracePeriod=10 Sep 16 17:34:38.894361 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.894331 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-8h852"] Sep 16 17:34:38.894661 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.894644 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce201115-8db1-4dde-b413-199ae2c9e693" containerName="console" Sep 16 17:34:38.894732 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.894663 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce201115-8db1-4dde-b413-199ae2c9e693" containerName="console" Sep 16 17:34:38.894781 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.894753 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce201115-8db1-4dde-b413-199ae2c9e693" containerName="console" Sep 16 17:34:38.896391 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.896373 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:38.913281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.913255 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-8h852"] Sep 16 17:34:38.997490 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.997457 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nb7\" (UniqueName: \"kubernetes.io/projected/8ca68af4-c999-4734-9cf3-22856a3ed8d6-kube-api-access-r4nb7\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:38.997622 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:38.997534 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ca68af4-c999-4734-9cf3-22856a3ed8d6-cert\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.097536 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.097514 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:34:39.097951 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.097930 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nb7\" (UniqueName: \"kubernetes.io/projected/8ca68af4-c999-4734-9cf3-22856a3ed8d6-kube-api-access-r4nb7\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.098021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.097994 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ca68af4-c999-4734-9cf3-22856a3ed8d6-cert\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.100094 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.100071 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ca68af4-c999-4734-9cf3-22856a3ed8d6-cert\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.107495 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.107473 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nb7\" (UniqueName: \"kubernetes.io/projected/8ca68af4-c999-4734-9cf3-22856a3ed8d6-kube-api-access-r4nb7\") pod \"kserve-controller-manager-cc8594966-8h852\" (UID: \"8ca68af4-c999-4734-9cf3-22856a3ed8d6\") " pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.199222 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.199139 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert\") pod \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " Sep 16 17:34:39.199222 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.199181 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dldld\" (UniqueName: \"kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld\") pod \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\" (UID: \"3a09c12e-a243-48f1-a0ac-f3e9a200948f\") " Sep 16 17:34:39.201331 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.201298 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert" (OuterVolumeSpecName: "cert") pod "3a09c12e-a243-48f1-a0ac-f3e9a200948f" (UID: "3a09c12e-a243-48f1-a0ac-f3e9a200948f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:34:39.201430 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.201303 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld" (OuterVolumeSpecName: "kube-api-access-dldld") pod "3a09c12e-a243-48f1-a0ac-f3e9a200948f" (UID: "3a09c12e-a243-48f1-a0ac-f3e9a200948f"). InnerVolumeSpecName "kube-api-access-dldld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:34:39.224384 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.224359 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:39.300040 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.300015 2536 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a09c12e-a243-48f1-a0ac-f3e9a200948f-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:34:39.300040 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.300038 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldld\" (UniqueName: \"kubernetes.io/projected/3a09c12e-a243-48f1-a0ac-f3e9a200948f-kube-api-access-dldld\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:34:39.497266 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.497235 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-8h852"] Sep 16 17:34:39.500347 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:34:39.500319 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca68af4_c999_4734_9cf3_22856a3ed8d6.slice/crio-ecb72c6e77c02bb75e0bea3ebb3214a2d1463af61c0f189c71da234e3a14e715 WatchSource:0}: Error finding container ecb72c6e77c02bb75e0bea3ebb3214a2d1463af61c0f189c71da234e3a14e715: Status 404 returned error can't find the container with id ecb72c6e77c02bb75e0bea3ebb3214a2d1463af61c0f189c71da234e3a14e715 Sep 16 17:34:39.852579 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.852554 2536 generic.go:356] "Generic (PLEG): container finished" podID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" containerID="0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909" exitCode=0 Sep 16 17:34:39.852682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.852614 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" Sep 16 17:34:39.852682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.852633 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" event={"ID":"3a09c12e-a243-48f1-a0ac-f3e9a200948f","Type":"ContainerDied","Data":"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909"} Sep 16 17:34:39.852779 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.852697 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-cqpxz" event={"ID":"3a09c12e-a243-48f1-a0ac-f3e9a200948f","Type":"ContainerDied","Data":"21f3e59e5343ee8597327e7f7530fd877635f251c6f850a0f67f7193b7690c56"} Sep 16 17:34:39.852779 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.852726 2536 scope.go:117] "RemoveContainer" containerID="0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909" Sep 16 17:34:39.854069 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.854034 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-8h852" event={"ID":"8ca68af4-c999-4734-9cf3-22856a3ed8d6","Type":"ContainerStarted","Data":"ecb72c6e77c02bb75e0bea3ebb3214a2d1463af61c0f189c71da234e3a14e715"} Sep 16 17:34:39.860851 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.860832 2536 scope.go:117] "RemoveContainer" containerID="0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909" Sep 16 17:34:39.861241 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:34:39.861219 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909\": container with ID starting with 0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909 not found: ID does not exist" containerID="0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909" Sep 16 17:34:39.861316 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.861250 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909"} err="failed to get container status \"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909\": rpc error: code = NotFound desc = could not find container \"0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909\": container with ID starting with 0d6e5232a101f208d088efc0ed23fb7a58329accf83f8a4dafacd7af44e08909 not found: ID does not exist" Sep 16 17:34:39.895868 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.895843 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:34:39.900006 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:39.899978 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-cqpxz"] Sep 16 17:34:40.477494 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:40.477459 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" path="/var/lib/kubelet/pods/3a09c12e-a243-48f1-a0ac-f3e9a200948f/volumes" Sep 16 17:34:40.859820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:40.859723 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-8h852" event={"ID":"8ca68af4-c999-4734-9cf3-22856a3ed8d6","Type":"ContainerStarted","Data":"7af2f017c30674d12ab196b77a85ea37b4fa1863ffb73159a33ce3dd1791c5ca"} Sep 16 17:34:40.860015 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:40.859861 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:34:40.883777 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:40.883729 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-8h852" podStartSLOduration=2.539404743 podStartE2EDuration="2.883713284s" totalImagesPullingTime="344.308541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:40.882294575 +0000 UTC m=+588.825616187" watchObservedRunningTime="2026-09-16 17:34:40.883713284 +0000 UTC m=+588.827034886" Sep 16 17:34:52.426485 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:52.426459 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:34:52.426887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:34:52.426861 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:35:11.868679 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:11.868649 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-8h852" Sep 16 17:35:13.470160 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.470121 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-8qkgz"] Sep 16 17:35:13.470551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.470413 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" containerName="manager" Sep 16 17:35:13.470551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.470424 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" containerName="manager" Sep 16 17:35:13.470551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.470474 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a09c12e-a243-48f1-a0ac-f3e9a200948f" containerName="manager" Sep 16 17:35:13.472153 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.472137 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.474986 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.474966 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5fl2m\"" Sep 16 17:35:13.475146 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.475127 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 17:35:13.486574 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.486546 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-8qkgz"] Sep 16 17:35:13.492708 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.492680 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-ncmrp"] Sep 16 17:35:13.494897 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.494882 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.497696 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.497676 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 16 17:35:13.497794 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.497714 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wcgdj\"" Sep 16 17:35:13.497972 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.497957 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 16 17:35:13.508996 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.508975 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ncmrp"] Sep 16 17:35:13.575083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.575044 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-cert\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.575083 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.575085 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj24t\" (UniqueName: \"kubernetes.io/projected/2177b553-aa4c-4840-a95e-4fa934e7565d-kube-api-access-bj24t\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.575334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.575117 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wmp\" (UniqueName: \"kubernetes.io/projected/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-kube-api-access-r8wmp\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.575334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.575212 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.575334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.575321 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-metrics-certs\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.675670 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.675640 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-cert\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.675850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.675684 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj24t\" (UniqueName: \"kubernetes.io/projected/2177b553-aa4c-4840-a95e-4fa934e7565d-kube-api-access-bj24t\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.675850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.675716 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wmp\" (UniqueName: \"kubernetes.io/projected/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-kube-api-access-r8wmp\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.675850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.675742 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.675850 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.675821 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-metrics-certs\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.676072 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:35:13.675953 2536 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 16 17:35:13.676072 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:35:13.676027 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs podName:2177b553-aa4c-4840-a95e-4fa934e7565d nodeName:}" failed. No retries permitted until 2026-09-16 17:35:14.176007732 +0000 UTC m=+622.119329313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs") pod "model-serving-api-7b954ffc95-8qkgz" (UID: "2177b553-aa4c-4840-a95e-4fa934e7565d") : secret "model-serving-api-tls" not found Sep 16 17:35:13.678368 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.678343 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-metrics-certs\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.678478 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.678343 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-cert\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.685889 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.685860 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj24t\" (UniqueName: \"kubernetes.io/projected/2177b553-aa4c-4840-a95e-4fa934e7565d-kube-api-access-bj24t\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:13.686269 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.686244 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wmp\" (UniqueName: \"kubernetes.io/projected/2b3b1702-cee2-4fc9-978c-96e2f0984ec5-kube-api-access-r8wmp\") pod \"odh-model-controller-7b966d7984-ncmrp\" (UID: \"2b3b1702-cee2-4fc9-978c-96e2f0984ec5\") " pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:13.804577 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:13.804492 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:14.087984 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.087896 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-ncmrp"] Sep 16 17:35:14.091507 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:35:14.091481 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3b1702_cee2_4fc9_978c_96e2f0984ec5.slice/crio-61a1c4802377274e0ad4de438906de6cebac5825084d8b01bdaf4a98fb57b7fb WatchSource:0}: Error finding container 61a1c4802377274e0ad4de438906de6cebac5825084d8b01bdaf4a98fb57b7fb: Status 404 returned error can't find the container with id 61a1c4802377274e0ad4de438906de6cebac5825084d8b01bdaf4a98fb57b7fb Sep 16 17:35:14.180352 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.180314 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:14.182691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.182671 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2177b553-aa4c-4840-a95e-4fa934e7565d-tls-certs\") pod \"model-serving-api-7b954ffc95-8qkgz\" (UID: \"2177b553-aa4c-4840-a95e-4fa934e7565d\") " pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:14.381672 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.381585 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:14.652056 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.651966 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-8qkgz"] Sep 16 17:35:14.655258 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:35:14.655225 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2177b553_aa4c_4840_a95e_4fa934e7565d.slice/crio-89edc425efbf8bd414e68420a6642689224767068bd7e8a386cb613bea0d994e WatchSource:0}: Error finding container 89edc425efbf8bd414e68420a6642689224767068bd7e8a386cb613bea0d994e: Status 404 returned error can't find the container with id 89edc425efbf8bd414e68420a6642689224767068bd7e8a386cb613bea0d994e Sep 16 17:35:14.998446 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.998406 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-8qkgz" event={"ID":"2177b553-aa4c-4840-a95e-4fa934e7565d","Type":"ContainerStarted","Data":"89edc425efbf8bd414e68420a6642689224767068bd7e8a386cb613bea0d994e"} Sep 16 17:35:14.999636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:14.999604 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ncmrp" event={"ID":"2b3b1702-cee2-4fc9-978c-96e2f0984ec5","Type":"ContainerStarted","Data":"61a1c4802377274e0ad4de438906de6cebac5825084d8b01bdaf4a98fb57b7fb"} Sep 16 17:35:18.016602 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.016556 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-ncmrp" event={"ID":"2b3b1702-cee2-4fc9-978c-96e2f0984ec5","Type":"ContainerStarted","Data":"24875a82fe56af96b04144589628c7c61d6e522c6545692baab452f8e94b81b8"} Sep 16 17:35:18.017198 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.016630 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:18.018312 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.018291 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-8qkgz" event={"ID":"2177b553-aa4c-4840-a95e-4fa934e7565d","Type":"ContainerStarted","Data":"c394b56fe2a8bc67c5c3e8e2746bd3a1141af111967a0220cd61bd3d86df8a84"} Sep 16 17:35:18.018468 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.018420 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:35:18.037299 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.037252 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-ncmrp" podStartSLOduration=1.5152268759999998 podStartE2EDuration="5.037237782s" totalImagesPullingTime="3.522010906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:18.035835881 +0000 UTC m=+625.979157492" watchObservedRunningTime="2026-09-16 17:35:18.037237782 +0000 UTC m=+625.980559413" Sep 16 17:35:18.056803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:18.056750 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-8qkgz" podStartSLOduration=2.089998939 podStartE2EDuration="5.056735795s" totalImagesPullingTime="2.966736856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:18.055673657 +0000 UTC m=+625.998995277" watchObservedRunningTime="2026-09-16 17:35:18.056735795 +0000 UTC m=+626.000057396" Sep 16 17:35:29.025171 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:29.025140 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-ncmrp" Sep 16 17:35:29.026773 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:35:29.026754 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-8qkgz" Sep 16 17:36:48.356461 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.356426 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:36:48.359520 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.359502 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.362079 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.362050 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Sep 16 17:36:48.368598 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.368575 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:36:48.460707 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.460661 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.460707 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.460711 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r722q\" (UniqueName: \"kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.561218 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.561184 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.561405 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.561234 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r722q\" (UniqueName: \"kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.561603 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.561578 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.570490 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.570461 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r722q\" (UniqueName: \"kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q\") pod \"seaweedfs-tls-custom-6644bf5d6c-dr9wg\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.668589 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.668500 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:36:48.936358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:48.936288 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:36:48.939007 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:36:48.938978 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97832d9_e86e_4668_aec9_0a0037fb119a.slice/crio-3d4cf733dba4565536d336e1c702bcbb6321c0a219658a2c770c82a91feccee4 WatchSource:0}: Error finding container 3d4cf733dba4565536d336e1c702bcbb6321c0a219658a2c770c82a91feccee4: Status 404 returned error can't find the container with id 3d4cf733dba4565536d336e1c702bcbb6321c0a219658a2c770c82a91feccee4 Sep 16 17:36:49.398346 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:49.398315 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" event={"ID":"b97832d9-e86e-4668-aec9-0a0037fb119a","Type":"ContainerStarted","Data":"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce"} Sep 16 17:36:49.398721 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:49.398365 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" event={"ID":"b97832d9-e86e-4668-aec9-0a0037fb119a","Type":"ContainerStarted","Data":"3d4cf733dba4565536d336e1c702bcbb6321c0a219658a2c770c82a91feccee4"} Sep 16 17:36:49.418008 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:49.417950 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" podStartSLOduration=1.160854957 podStartE2EDuration="1.417929844s" totalImagesPullingTime="257.074887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:49.417202138 +0000 UTC m=+717.360523741" watchObservedRunningTime="2026-09-16 17:36:49.417929844 +0000 UTC m=+717.361251456" Sep 16 17:36:50.535814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:50.535776 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:36:51.406808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:36:51.406766 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" podUID="b97832d9-e86e-4668-aec9-0a0037fb119a" containerName="seaweedfs-tls-custom" containerID="cri-o://7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce" gracePeriod=30 Sep 16 17:37:19.841334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:19.841311 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:37:20.026318 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.026281 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r722q\" (UniqueName: \"kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q\") pod \"b97832d9-e86e-4668-aec9-0a0037fb119a\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " Sep 16 17:37:20.026506 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.026378 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data\") pod \"b97832d9-e86e-4668-aec9-0a0037fb119a\" (UID: \"b97832d9-e86e-4668-aec9-0a0037fb119a\") " Sep 16 17:37:20.027683 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.027654 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data" (OuterVolumeSpecName: "data") pod "b97832d9-e86e-4668-aec9-0a0037fb119a" (UID: "b97832d9-e86e-4668-aec9-0a0037fb119a"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:37:20.028430 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.028402 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q" (OuterVolumeSpecName: "kube-api-access-r722q") pod "b97832d9-e86e-4668-aec9-0a0037fb119a" (UID: "b97832d9-e86e-4668-aec9-0a0037fb119a"). InnerVolumeSpecName "kube-api-access-r722q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:37:20.127244 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.127209 2536 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b97832d9-e86e-4668-aec9-0a0037fb119a-data\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:37:20.127244 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.127238 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r722q\" (UniqueName: \"kubernetes.io/projected/b97832d9-e86e-4668-aec9-0a0037fb119a-kube-api-access-r722q\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:37:20.525868 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.525835 2536 generic.go:356] "Generic (PLEG): container finished" podID="b97832d9-e86e-4668-aec9-0a0037fb119a" containerID="7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce" exitCode=0 Sep 16 17:37:20.526021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.525875 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" Sep 16 17:37:20.526021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.525889 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" event={"ID":"b97832d9-e86e-4668-aec9-0a0037fb119a","Type":"ContainerDied","Data":"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce"} Sep 16 17:37:20.526021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.525951 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg" event={"ID":"b97832d9-e86e-4668-aec9-0a0037fb119a","Type":"ContainerDied","Data":"3d4cf733dba4565536d336e1c702bcbb6321c0a219658a2c770c82a91feccee4"} Sep 16 17:37:20.526021 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.525978 2536 scope.go:117] "RemoveContainer" containerID="7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce" Sep 16 17:37:20.535127 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.535112 2536 scope.go:117] "RemoveContainer" containerID="7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce" Sep 16 17:37:20.535389 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:37:20.535362 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce\": container with ID starting with 7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce not found: ID does not exist" containerID="7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce" Sep 16 17:37:20.535429 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.535396 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce"} err="failed to get container status \"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce\": rpc error: code = NotFound desc = could not find container \"7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce\": container with ID starting with 7f87ae46331d534743d86890e9e6e0afc9c079a0e09c7b84f187bb11dc5669ce not found: ID does not exist" Sep 16 17:37:20.543029 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.543005 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:37:20.545385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.545365 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-dr9wg"] Sep 16 17:37:20.586559 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.586529 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cschg"] Sep 16 17:37:20.586846 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.586836 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97832d9-e86e-4668-aec9-0a0037fb119a" containerName="seaweedfs-tls-custom" Sep 16 17:37:20.586887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.586848 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97832d9-e86e-4668-aec9-0a0037fb119a" containerName="seaweedfs-tls-custom" Sep 16 17:37:20.586946 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.586910 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97832d9-e86e-4668-aec9-0a0037fb119a" containerName="seaweedfs-tls-custom" Sep 16 17:37:20.591212 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.591196 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.594122 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.594098 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Sep 16 17:37:20.594313 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.594298 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\"" Sep 16 17:37:20.599729 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.599706 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cschg"] Sep 16 17:37:20.732013 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.731972 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.732189 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.732034 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jhg\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-kube-api-access-s5jhg\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.732189 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.732055 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/257473ad-fae8-475a-857e-eaafcb9f77a0-data\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.832997 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.832898 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.832997 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.832978 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jhg\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-kube-api-access-s5jhg\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.833173 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.833000 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/257473ad-fae8-475a-857e-eaafcb9f77a0-data\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.833382 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.833359 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/257473ad-fae8-475a-857e-eaafcb9f77a0-data\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.835216 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.835196 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.841543 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.841521 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jhg\" (UniqueName: \"kubernetes.io/projected/257473ad-fae8-475a-857e-eaafcb9f77a0-kube-api-access-s5jhg\") pod \"seaweedfs-tls-custom-8b44d767d-cschg\" (UID: \"257473ad-fae8-475a-857e-eaafcb9f77a0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:20.900779 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:20.900751 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" Sep 16 17:37:21.189509 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:21.189464 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cschg"] Sep 16 17:37:21.191588 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:37:21.191552 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257473ad_fae8_475a_857e_eaafcb9f77a0.slice/crio-3b8a62e29fae0c1808fab8b850b36c959961966fbe93402afce4c60b2c92e956 WatchSource:0}: Error finding container 3b8a62e29fae0c1808fab8b850b36c959961966fbe93402afce4c60b2c92e956: Status 404 returned error can't find the container with id 3b8a62e29fae0c1808fab8b850b36c959961966fbe93402afce4c60b2c92e956 Sep 16 17:37:21.193011 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:21.192984 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 17:37:21.532784 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:21.532753 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" event={"ID":"257473ad-fae8-475a-857e-eaafcb9f77a0","Type":"ContainerStarted","Data":"5026f6551347d40157dd767d2f3ff85f274ad890b586fa8f47764df8f86334ca"} Sep 16 17:37:21.532971 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:21.532806 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" event={"ID":"257473ad-fae8-475a-857e-eaafcb9f77a0","Type":"ContainerStarted","Data":"3b8a62e29fae0c1808fab8b850b36c959961966fbe93402afce4c60b2c92e956"} Sep 16 17:37:21.551970 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:21.551884 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-cschg" podStartSLOduration=1.290354725 podStartE2EDuration="1.551864808s" totalImagesPullingTime="261.510083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:21.550512659 +0000 UTC m=+749.493834285" watchObservedRunningTime="2026-09-16 17:37:21.551864808 +0000 UTC m=+749.495186411" Sep 16 17:37:22.477891 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:22.477858 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97832d9-e86e-4668-aec9-0a0037fb119a" path="/var/lib/kubelet/pods/b97832d9-e86e-4668-aec9-0a0037fb119a/volumes" Sep 16 17:37:34.450447 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.450414 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h59vg"] Sep 16 17:37:34.453756 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.453739 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.456327 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.456307 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\"" Sep 16 17:37:34.456946 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.456930 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Sep 16 17:37:34.467673 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.467647 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h59vg"] Sep 16 17:37:34.539157 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.539114 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.539157 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.539159 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.539365 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.539205 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmpk\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-kube-api-access-jnmpk\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.639772 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.639737 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmpk\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-kube-api-access-jnmpk\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.639975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.639805 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.639975 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.639821 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.639975 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:37:34.639893 2536 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found Sep 16 17:37:34.639975 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:37:34.639936 2536 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-h59vg: secret "seaweedfs-tls-serving" not found Sep 16 17:37:34.640130 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:37:34.640006 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving podName:a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68 nodeName:}" failed. No retries permitted until 2026-09-16 17:37:35.139990003 +0000 UTC m=+763.083311584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-h59vg" (UID: "a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68") : secret "seaweedfs-tls-serving" not found Sep 16 17:37:34.640188 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.640174 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:34.651624 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:34.651593 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmpk\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-kube-api-access-jnmpk\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:35.144200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:35.144168 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:35.146600 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:35.146576 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h59vg\" (UID: \"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:35.363255 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:35.363212 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" Sep 16 17:37:35.632591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:35.632509 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h59vg"] Sep 16 17:37:35.635271 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:37:35.635245 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63a44b2_ed7e_44a3_bfa4_c994ea6ddf68.slice/crio-4fa4e2bd1db34f5184a287d7214f9212e4292399175d9bab61359f15c626fd0d WatchSource:0}: Error finding container 4fa4e2bd1db34f5184a287d7214f9212e4292399175d9bab61359f15c626fd0d: Status 404 returned error can't find the container with id 4fa4e2bd1db34f5184a287d7214f9212e4292399175d9bab61359f15c626fd0d Sep 16 17:37:36.598361 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:36.598322 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" event={"ID":"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68","Type":"ContainerStarted","Data":"765e4b1219534335443ac769ad36adf487cdcfd196ebed770bb4a9dab80bae23"} Sep 16 17:37:36.598539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:36.598381 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" event={"ID":"a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68","Type":"ContainerStarted","Data":"4fa4e2bd1db34f5184a287d7214f9212e4292399175d9bab61359f15c626fd0d"} Sep 16 17:37:36.615522 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:37:36.615475 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h59vg" podStartSLOduration=2.331837715 podStartE2EDuration="2.615458581s" totalImagesPullingTime="283.620866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:36.614558115 +0000 UTC m=+764.557879741" watchObservedRunningTime="2026-09-16 17:37:36.615458581 +0000 UTC m=+764.558780183" Sep 16 17:39:52.451445 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:39:52.451414 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:39:52.451898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:39:52.451799 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:41:12.059523 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.059487 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:41:12.062792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.062775 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.065741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.065702 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:41:12.065741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.065725 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\"" Sep 16 17:41:12.065741 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.065702 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\"" Sep 16 17:41:12.065999 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.065738 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:41:12.066712 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.066692 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:41:12.072637 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.072612 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:41:12.154553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.154521 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.154716 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.154572 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46g9m\" (UniqueName: \"kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.154716 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.154612 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.255818 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.255786 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46g9m\" (UniqueName: \"kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.255996 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.255828 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.255996 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.255893 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.256512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.256484 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.258285 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.258254 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.265318 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.265298 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46g9m\" (UniqueName: \"kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m\") pod \"message-dumper-predictor-6575bcfdfd-qcmdr\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.374455 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.374367 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:12.658725 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:12.658697 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:41:12.661312 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:41:12.661284 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76df728_433a_401c_824d_72b8677d42c3.slice/crio-df51a371d1ef0519d3fdf29cfd41403e22dfc5f30559497eb5a6b865950bd7e4 WatchSource:0}: Error finding container df51a371d1ef0519d3fdf29cfd41403e22dfc5f30559497eb5a6b865950bd7e4: Status 404 returned error can't find the container with id df51a371d1ef0519d3fdf29cfd41403e22dfc5f30559497eb5a6b865950bd7e4 Sep 16 17:41:13.525267 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:13.525227 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerStarted","Data":"df51a371d1ef0519d3fdf29cfd41403e22dfc5f30559497eb5a6b865950bd7e4"} Sep 16 17:41:14.533399 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:14.533357 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerStarted","Data":"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241"} Sep 16 17:41:16.544704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:16.544671 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerStarted","Data":"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef"} Sep 16 17:41:16.545185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:16.544936 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:16.545185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:16.545091 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:16.546943 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:16.546909 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:41:16.563557 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:16.563515 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" podStartSLOduration=1.223053124 podStartE2EDuration="4.563500531s" totalImagesPullingTime="3.340447407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:16.562370633 +0000 UTC m=+984.505692239" watchObservedRunningTime="2026-09-16 17:41:16.563500531 +0000 UTC m=+984.506822135" Sep 16 17:41:23.558196 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:41:23.558169 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:42:57.139687 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.139655 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-qcmdr_a76df728-433a-401c-824d-72b8677d42c3/kserve-container/0.log" Sep 16 17:42:57.355155 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.355124 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:42:57.355376 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.355352 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kserve-container" containerID="cri-o://ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" gracePeriod=30 Sep 16 17:42:57.355462 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.355400 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kube-rbac-proxy" containerID="cri-o://c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" gracePeriod=30 Sep 16 17:42:57.598399 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.598375 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:42:57.705982 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.705942 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config\") pod \"a76df728-433a-401c-824d-72b8677d42c3\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " Sep 16 17:42:57.706170 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.705990 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls\") pod \"a76df728-433a-401c-824d-72b8677d42c3\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " Sep 16 17:42:57.706170 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.706094 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46g9m\" (UniqueName: \"kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m\") pod \"a76df728-433a-401c-824d-72b8677d42c3\" (UID: \"a76df728-433a-401c-824d-72b8677d42c3\") " Sep 16 17:42:57.706328 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.706303 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a76df728-433a-401c-824d-72b8677d42c3" (UID: "a76df728-433a-401c-824d-72b8677d42c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:42:57.708286 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.708258 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m" (OuterVolumeSpecName: "kube-api-access-46g9m") pod "a76df728-433a-401c-824d-72b8677d42c3" (UID: "a76df728-433a-401c-824d-72b8677d42c3"). InnerVolumeSpecName "kube-api-access-46g9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:42:57.708286 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.708272 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a76df728-433a-401c-824d-72b8677d42c3" (UID: "a76df728-433a-401c-824d-72b8677d42c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:42:57.807591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.807557 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a76df728-433a-401c-824d-72b8677d42c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:42:57.807591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.807585 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a76df728-433a-401c-824d-72b8677d42c3-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:42:57.807591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.807594 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46g9m\" (UniqueName: \"kubernetes.io/projected/a76df728-433a-401c-824d-72b8677d42c3-kube-api-access-46g9m\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:42:57.985721 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985690 2536 generic.go:356] "Generic (PLEG): container finished" podID="a76df728-433a-401c-824d-72b8677d42c3" containerID="c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" exitCode=2 Sep 16 17:42:57.985721 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985716 2536 generic.go:356] "Generic (PLEG): container finished" podID="a76df728-433a-401c-824d-72b8677d42c3" containerID="ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" exitCode=2 Sep 16 17:42:57.985911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985741 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" Sep 16 17:42:57.985911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985760 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerDied","Data":"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef"} Sep 16 17:42:57.985911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985812 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerDied","Data":"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241"} Sep 16 17:42:57.985911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985831 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr" event={"ID":"a76df728-433a-401c-824d-72b8677d42c3","Type":"ContainerDied","Data":"df51a371d1ef0519d3fdf29cfd41403e22dfc5f30559497eb5a6b865950bd7e4"} Sep 16 17:42:57.985911 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.985854 2536 scope.go:117] "RemoveContainer" containerID="c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" Sep 16 17:42:57.995031 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:57.995012 2536 scope.go:117] "RemoveContainer" containerID="ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" Sep 16 17:42:58.002431 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.002415 2536 scope.go:117] "RemoveContainer" containerID="c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" Sep 16 17:42:58.002693 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:42:58.002674 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef\": container with ID starting with c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef not found: ID does not exist" containerID="c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" Sep 16 17:42:58.002749 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.002701 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef"} err="failed to get container status \"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef\": rpc error: code = NotFound desc = could not find container \"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef\": container with ID starting with c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef not found: ID does not exist" Sep 16 17:42:58.002749 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.002717 2536 scope.go:117] "RemoveContainer" containerID="ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" Sep 16 17:42:58.002977 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:42:58.002963 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241\": container with ID starting with ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241 not found: ID does not exist" containerID="ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" Sep 16 17:42:58.003029 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.002980 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241"} err="failed to get container status \"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241\": rpc error: code = NotFound desc = could not find container \"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241\": container with ID starting with ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241 not found: ID does not exist" Sep 16 17:42:58.003029 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.002992 2536 scope.go:117] "RemoveContainer" containerID="c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef" Sep 16 17:42:58.003226 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.003207 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef"} err="failed to get container status \"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef\": rpc error: code = NotFound desc = could not find container \"c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef\": container with ID starting with c91044e48863a853aa24020f646618eeb5fee4716f36810c09e2d6add32f5eef not found: ID does not exist" Sep 16 17:42:58.003266 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.003228 2536 scope.go:117] "RemoveContainer" containerID="ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241" Sep 16 17:42:58.003468 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.003452 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241"} err="failed to get container status \"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241\": rpc error: code = NotFound desc = could not find container \"ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241\": container with ID starting with ca8be29244e3d7d38c6919244ac95a5b69c494d5ed825d0d773cbdf18c43e241 not found: ID does not exist" Sep 16 17:42:58.008332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.008310 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:42:58.011903 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.011882 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qcmdr"] Sep 16 17:42:58.477709 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:42:58.477668 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76df728-433a-401c-824d-72b8677d42c3" path="/var/lib/kubelet/pods/a76df728-433a-401c-824d-72b8677d42c3/volumes" Sep 16 17:43:08.133688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.133653 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:43:08.134101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134070 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kserve-container" Sep 16 17:43:08.134101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134084 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kserve-container" Sep 16 17:43:08.134101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134093 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kube-rbac-proxy" Sep 16 17:43:08.134101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134099 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kube-rbac-proxy" Sep 16 17:43:08.134257 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134149 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kube-rbac-proxy" Sep 16 17:43:08.134257 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.134160 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76df728-433a-401c-824d-72b8677d42c3" containerName="kserve-container" Sep 16 17:43:08.138608 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.138590 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.146039 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.146019 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\"" Sep 16 17:43:08.146158 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.146038 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\"" Sep 16 17:43:08.146158 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.146076 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:43:08.146267 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.146193 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:43:08.147217 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.147203 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:43:08.147479 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.147464 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:43:08.147541 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.147524 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:43:08.159269 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.159242 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:43:08.195880 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.195851 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bhv\" (UniqueName: \"kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.196052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.195890 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.196052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.195959 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.196052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.195977 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.196052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.196036 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296438 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bhv\" (UniqueName: \"kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296479 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296675 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296514 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296675 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296533 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296675 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296553 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.296964 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.296938 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.297292 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.297271 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.297372 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.297274 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.299085 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.299064 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.305580 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.305552 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bhv\" (UniqueName: \"kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.448658 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.448572 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:08.723098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.723070 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:43:08.724950 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:43:08.724923 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104b09f0_9792_4e05_9d50_5200d515c268.slice/crio-d2c50facc4752e54c4f619f18adadb5e250b268c0e9b935408b988a503330951 WatchSource:0}: Error finding container d2c50facc4752e54c4f619f18adadb5e250b268c0e9b935408b988a503330951: Status 404 returned error can't find the container with id d2c50facc4752e54c4f619f18adadb5e250b268c0e9b935408b988a503330951 Sep 16 17:43:08.726963 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:08.726946 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 17:43:09.033912 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:09.033829 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerStarted","Data":"d2c50facc4752e54c4f619f18adadb5e250b268c0e9b935408b988a503330951"} Sep 16 17:43:12.050689 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:12.050650 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerStarted","Data":"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6"} Sep 16 17:43:16.070435 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:16.070401 2536 generic.go:356] "Generic (PLEG): container finished" podID="104b09f0-9792-4e05-9d50-5200d515c268" containerID="0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6" exitCode=0 Sep 16 17:43:16.070815 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:16.070481 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerDied","Data":"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6"} Sep 16 17:43:37.192676 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:37.192643 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerStarted","Data":"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b"} Sep 16 17:43:38.199578 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:38.199536 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerStarted","Data":"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d"} Sep 16 17:43:38.199959 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:38.199755 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:38.223756 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:38.223690 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podStartSLOduration=2.459510625 podStartE2EDuration="30.223674974s" totalImagesPullingTime="24.202582085s" totalInitContainerRuntime="3.561582264s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:43:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:43:38.222078428 +0000 UTC m=+1126.165400048" watchObservedRunningTime="2026-09-16 17:43:38.223674974 +0000 UTC m=+1126.166996576" Sep 16 17:43:39.204439 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:39.204402 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:43:45.215539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:43:45.215511 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:44:15.219049 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:15.218934 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 17:44:25.216572 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:25.216530 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 17:44:35.216745 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:35.216708 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 17:44:45.216262 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:45.216224 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 17:44:52.478382 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:52.478354 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:44:52.480548 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:52.480528 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:44:53.493606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:53.493568 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:44:58.251355 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:58.251322 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:44:58.251887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:58.251601 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" containerID="cri-o://e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b" gracePeriod=30 Sep 16 17:44:58.251887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:58.251683 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kube-rbac-proxy" containerID="cri-o://89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d" gracePeriod=30 Sep 16 17:44:58.551167 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:58.551087 2536 generic.go:356] "Generic (PLEG): container finished" podID="104b09f0-9792-4e05-9d50-5200d515c268" containerID="89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d" exitCode=2 Sep 16 17:44:58.551167 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:44:58.551153 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerDied","Data":"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d"} Sep 16 17:45:00.210578 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:00.210539 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused" Sep 16 17:45:03.090066 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.090039 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:45:03.181391 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181301 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location\") pod \"104b09f0-9792-4e05-9d50-5200d515c268\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " Sep 16 17:45:03.181391 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181364 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6bhv\" (UniqueName: \"kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv\") pod \"104b09f0-9792-4e05-9d50-5200d515c268\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " Sep 16 17:45:03.181601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181410 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert\") pod \"104b09f0-9792-4e05-9d50-5200d515c268\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " Sep 16 17:45:03.181601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181438 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls\") pod \"104b09f0-9792-4e05-9d50-5200d515c268\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " Sep 16 17:45:03.181601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181484 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config\") pod \"104b09f0-9792-4e05-9d50-5200d515c268\" (UID: \"104b09f0-9792-4e05-9d50-5200d515c268\") " Sep 16 17:45:03.181803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181689 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "104b09f0-9792-4e05-9d50-5200d515c268" (UID: "104b09f0-9792-4e05-9d50-5200d515c268"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:45:03.181803 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181772 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "104b09f0-9792-4e05-9d50-5200d515c268" (UID: "104b09f0-9792-4e05-9d50-5200d515c268"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:45:03.182011 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.181903 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "104b09f0-9792-4e05-9d50-5200d515c268" (UID: "104b09f0-9792-4e05-9d50-5200d515c268"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:45:03.183659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.183630 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv" (OuterVolumeSpecName: "kube-api-access-l6bhv") pod "104b09f0-9792-4e05-9d50-5200d515c268" (UID: "104b09f0-9792-4e05-9d50-5200d515c268"). InnerVolumeSpecName "kube-api-access-l6bhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:45:03.183659 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.183638 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "104b09f0-9792-4e05-9d50-5200d515c268" (UID: "104b09f0-9792-4e05-9d50-5200d515c268"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:45:03.282358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.282322 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:45:03.282358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.282352 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b09f0-9792-4e05-9d50-5200d515c268-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:45:03.282358 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.282361 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b09f0-9792-4e05-9d50-5200d515c268-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:45:03.282576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.282370 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104b09f0-9792-4e05-9d50-5200d515c268-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:45:03.282576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.282379 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6bhv\" (UniqueName: \"kubernetes.io/projected/104b09f0-9792-4e05-9d50-5200d515c268-kube-api-access-l6bhv\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:45:03.426227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426193 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:45:03.426512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426500 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kube-rbac-proxy" Sep 16 17:45:03.426553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426515 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kube-rbac-proxy" Sep 16 17:45:03.426553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426525 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" Sep 16 17:45:03.426553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426531 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" Sep 16 17:45:03.426553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426539 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="storage-initializer" Sep 16 17:45:03.426553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426545 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="storage-initializer" Sep 16 17:45:03.426691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426600 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kube-rbac-proxy" Sep 16 17:45:03.426691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.426608 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="104b09f0-9792-4e05-9d50-5200d515c268" containerName="kserve-container" Sep 16 17:45:03.428648 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.428627 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.431496 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.431439 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\"" Sep 16 17:45:03.431596 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.431579 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\"" Sep 16 17:45:03.442618 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.442593 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:45:03.484168 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.484133 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.484168 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.484179 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcjh\" (UniqueName: \"kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.484388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.484202 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.484388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.484278 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.484388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.484304 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.573278 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.573252 2536 generic.go:356] "Generic (PLEG): container finished" podID="104b09f0-9792-4e05-9d50-5200d515c268" containerID="e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b" exitCode=0 Sep 16 17:45:03.573437 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.573315 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" Sep 16 17:45:03.573437 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.573338 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerDied","Data":"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b"} Sep 16 17:45:03.573437 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.573394 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m" event={"ID":"104b09f0-9792-4e05-9d50-5200d515c268","Type":"ContainerDied","Data":"d2c50facc4752e54c4f619f18adadb5e250b268c0e9b935408b988a503330951"} Sep 16 17:45:03.573437 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.573420 2536 scope.go:117] "RemoveContainer" containerID="89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d" Sep 16 17:45:03.582211 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.582190 2536 scope.go:117] "RemoveContainer" containerID="e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b" Sep 16 17:45:03.585544 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.585523 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.585645 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.585560 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.585645 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.585584 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.585645 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.585617 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcjh\" (UniqueName: \"kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.585645 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.585646 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.586043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.586009 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.586439 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.586406 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.586530 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.586406 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.587857 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.587839 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.590237 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.590223 2536 scope.go:117] "RemoveContainer" containerID="0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6" Sep 16 17:45:03.594076 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.594055 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcjh\" (UniqueName: \"kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:03.596808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.596788 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:45:03.600414 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.600394 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-v8b5m"] Sep 16 17:45:03.600680 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.600661 2536 scope.go:117] "RemoveContainer" containerID="89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d" Sep 16 17:45:03.600979 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:45:03.600962 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d\": container with ID starting with 89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d not found: ID does not exist" containerID="89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d" Sep 16 17:45:03.601040 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.600986 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d"} err="failed to get container status \"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d\": rpc error: code = NotFound desc = could not find container \"89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d\": container with ID starting with 89a74745863f8d3c9518f4271aa6aa80cfc70212404f1fbacb044ae48182a42d not found: ID does not exist" Sep 16 17:45:03.601040 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.601003 2536 scope.go:117] "RemoveContainer" containerID="e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b" Sep 16 17:45:03.601254 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:45:03.601232 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b\": container with ID starting with e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b not found: ID does not exist" containerID="e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b" Sep 16 17:45:03.601331 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.601258 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b"} err="failed to get container status \"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b\": rpc error: code = NotFound desc = could not find container \"e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b\": container with ID starting with e75a8135344c1ae35d4d9bbb28b7a82285f2c3643e128ec958387fa6a2d2fd4b not found: ID does not exist" Sep 16 17:45:03.601331 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.601275 2536 scope.go:117] "RemoveContainer" containerID="0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6" Sep 16 17:45:03.601508 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:45:03.601492 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6\": container with ID starting with 0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6 not found: ID does not exist" containerID="0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6" Sep 16 17:45:03.601559 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.601515 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6"} err="failed to get container status \"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6\": rpc error: code = NotFound desc = could not find container \"0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6\": container with ID starting with 0c9518f1358b7eda65c4154370ca65e4b92795559789b3aadb0c9366925b35b6 not found: ID does not exist" Sep 16 17:45:03.739408 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:03.739372 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:04.028696 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:04.028373 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:45:04.031108 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:45:04.031079 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffccdf2_43df_4410_b116_c5519137411a.slice/crio-e33639464d574ae3554c970c1b03348e00dc74984647d3e88db9ab27c1626c54 WatchSource:0}: Error finding container e33639464d574ae3554c970c1b03348e00dc74984647d3e88db9ab27c1626c54: Status 404 returned error can't find the container with id e33639464d574ae3554c970c1b03348e00dc74984647d3e88db9ab27c1626c54 Sep 16 17:45:04.477503 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:04.477464 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104b09f0-9792-4e05-9d50-5200d515c268" path="/var/lib/kubelet/pods/104b09f0-9792-4e05-9d50-5200d515c268/volumes" Sep 16 17:45:04.580632 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:04.580599 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerStarted","Data":"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999"} Sep 16 17:45:04.580800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:04.580660 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerStarted","Data":"e33639464d574ae3554c970c1b03348e00dc74984647d3e88db9ab27c1626c54"} Sep 16 17:45:08.598464 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:08.598425 2536 generic.go:356] "Generic (PLEG): container finished" podID="effccdf2-43df-4410-b116-c5519137411a" containerID="7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999" exitCode=0 Sep 16 17:45:08.598843 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:08.598468 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerDied","Data":"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999"} Sep 16 17:45:09.605052 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:09.605018 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerStarted","Data":"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4"} Sep 16 17:45:09.605513 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:09.605087 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerStarted","Data":"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7"} Sep 16 17:45:09.605513 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:09.605300 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:09.629700 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:09.629649 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podStartSLOduration=2.723363432 podStartE2EDuration="6.629633144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906269712s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:45:09.62925676 +0000 UTC m=+1217.572578386" watchObservedRunningTime="2026-09-16 17:45:09.629633144 +0000 UTC m=+1217.572954745" Sep 16 17:45:10.609657 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:10.609624 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:16.619802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:16.619772 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:45:46.620582 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:46.620502 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 17:45:56.620849 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:56.620811 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 17:45:58.871784 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.871744 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:45:58.876448 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.876426 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:58.879900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.879878 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\"" Sep 16 17:45:58.880270 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.880249 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\"" Sep 16 17:45:58.892667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.892640 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:45:58.943537 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.943502 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:58.943537 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.943540 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:58.943775 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.943567 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:58.943775 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.943673 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmkl\" (UniqueName: \"kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:58.943775 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:58.943709 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.044609 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.044572 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.044800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.044622 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.044800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.044658 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.044800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.044698 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmkl\" (UniqueName: \"kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.044800 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.044721 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.045051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.045011 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.045364 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.045342 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.045438 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.045361 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.047027 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.047009 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.056070 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.056049 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmkl\" (UniqueName: \"kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-22dml\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.186114 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.186029 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:45:59.464037 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.464012 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:45:59.465745 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:45:59.465719 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e09ebd3_183a_4916_9140_1b263ba226f9.slice/crio-3d3a5490b9d06449e05599355b20159203fba60439a96f123ce35ee4ace0f331 WatchSource:0}: Error finding container 3d3a5490b9d06449e05599355b20159203fba60439a96f123ce35ee4ace0f331: Status 404 returned error can't find the container with id 3d3a5490b9d06449e05599355b20159203fba60439a96f123ce35ee4ace0f331 Sep 16 17:45:59.830373 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.830336 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerStarted","Data":"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd"} Sep 16 17:45:59.830560 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:45:59.830416 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerStarted","Data":"3d3a5490b9d06449e05599355b20159203fba60439a96f123ce35ee4ace0f331"} Sep 16 17:46:05.860291 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:05.860253 2536 generic.go:356] "Generic (PLEG): container finished" podID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerID="720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd" exitCode=0 Sep 16 17:46:05.860685 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:05.860324 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerDied","Data":"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd"} Sep 16 17:46:06.620479 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:06.620437 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 17:46:16.621342 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:16.621290 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 17:46:26.694820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:26.694782 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:46:33.500853 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:33.500813 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:46:33.501361 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:33.501051 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" containerID="cri-o://7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7" gracePeriod=30 Sep 16 17:46:33.501361 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:33.501098 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kube-rbac-proxy" containerID="cri-o://af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4" gracePeriod=30 Sep 16 17:46:34.005051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:34.005014 2536 generic.go:356] "Generic (PLEG): container finished" podID="effccdf2-43df-4410-b116-c5519137411a" containerID="af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4" exitCode=2 Sep 16 17:46:34.005232 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:34.005087 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerDied","Data":"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4"} Sep 16 17:46:36.614798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:36.614752 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused" Sep 16 17:46:36.621273 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:36.621227 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 17:46:38.847575 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.847551 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:46:38.894309 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894264 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfcjh\" (UniqueName: \"kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh\") pod \"effccdf2-43df-4410-b116-c5519137411a\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " Sep 16 17:46:38.894518 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894366 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert\") pod \"effccdf2-43df-4410-b116-c5519137411a\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " Sep 16 17:46:38.894518 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894391 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config\") pod \"effccdf2-43df-4410-b116-c5519137411a\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " Sep 16 17:46:38.894518 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894413 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location\") pod \"effccdf2-43df-4410-b116-c5519137411a\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " Sep 16 17:46:38.894518 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894489 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls\") pod \"effccdf2-43df-4410-b116-c5519137411a\" (UID: \"effccdf2-43df-4410-b116-c5519137411a\") " Sep 16 17:46:38.894793 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894767 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "effccdf2-43df-4410-b116-c5519137411a" (UID: "effccdf2-43df-4410-b116-c5519137411a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:46:38.894974 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894798 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "effccdf2-43df-4410-b116-c5519137411a" (UID: "effccdf2-43df-4410-b116-c5519137411a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:46:38.894974 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.894824 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "effccdf2-43df-4410-b116-c5519137411a" (UID: "effccdf2-43df-4410-b116-c5519137411a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:46:38.896598 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.896572 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh" (OuterVolumeSpecName: "kube-api-access-rfcjh") pod "effccdf2-43df-4410-b116-c5519137411a" (UID: "effccdf2-43df-4410-b116-c5519137411a"). InnerVolumeSpecName "kube-api-access-rfcjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:46:38.896795 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.896776 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "effccdf2-43df-4410-b116-c5519137411a" (UID: "effccdf2-43df-4410-b116-c5519137411a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:46:38.995743 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.995705 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/effccdf2-43df-4410-b116-c5519137411a-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:46:38.995743 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.995739 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcjh\" (UniqueName: \"kubernetes.io/projected/effccdf2-43df-4410-b116-c5519137411a-kube-api-access-rfcjh\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:46:38.995977 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.995755 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:46:38.995977 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.995765 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/effccdf2-43df-4410-b116-c5519137411a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:46:38.995977 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:38.995774 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/effccdf2-43df-4410-b116-c5519137411a-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:46:39.033337 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.033248 2536 generic.go:356] "Generic (PLEG): container finished" podID="effccdf2-43df-4410-b116-c5519137411a" containerID="7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7" exitCode=0 Sep 16 17:46:39.033337 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.033314 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" Sep 16 17:46:39.033551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.033314 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerDied","Data":"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7"} Sep 16 17:46:39.033551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.033462 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z" event={"ID":"effccdf2-43df-4410-b116-c5519137411a","Type":"ContainerDied","Data":"e33639464d574ae3554c970c1b03348e00dc74984647d3e88db9ab27c1626c54"} Sep 16 17:46:39.033551 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.033506 2536 scope.go:117] "RemoveContainer" containerID="af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4" Sep 16 17:46:39.044051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.043877 2536 scope.go:117] "RemoveContainer" containerID="7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7" Sep 16 17:46:39.052941 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.052909 2536 scope.go:117] "RemoveContainer" containerID="7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999" Sep 16 17:46:39.061679 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.061660 2536 scope.go:117] "RemoveContainer" containerID="af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4" Sep 16 17:46:39.062063 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:46:39.062038 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4\": container with ID starting with af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4 not found: ID does not exist" containerID="af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4" Sep 16 17:46:39.062155 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.062071 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4"} err="failed to get container status \"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4\": rpc error: code = NotFound desc = could not find container \"af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4\": container with ID starting with af9c8ee792379003e70e01a2eca83b2c894b2e536ffca2ff5351f4d564d640d4 not found: ID does not exist" Sep 16 17:46:39.062155 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.062090 2536 scope.go:117] "RemoveContainer" containerID="7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7" Sep 16 17:46:39.062398 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:46:39.062373 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7\": container with ID starting with 7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7 not found: ID does not exist" containerID="7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7" Sep 16 17:46:39.062510 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.062408 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7"} err="failed to get container status \"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7\": rpc error: code = NotFound desc = could not find container \"7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7\": container with ID starting with 7f9f544d4141a33889821b287d82ec735981e57c3f2784a79c709006aa3af0f7 not found: ID does not exist" Sep 16 17:46:39.062510 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.062432 2536 scope.go:117] "RemoveContainer" containerID="7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999" Sep 16 17:46:39.062879 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:46:39.062853 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999\": container with ID starting with 7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999 not found: ID does not exist" containerID="7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999" Sep 16 17:46:39.062996 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.062887 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999"} err="failed to get container status \"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999\": rpc error: code = NotFound desc = could not find container \"7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999\": container with ID starting with 7980b96f4c9a78a69e9b2e2700304b0e69229ccd2701fd0b646ff54f625a0999 not found: ID does not exist" Sep 16 17:46:39.064240 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.064130 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:46:39.066472 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:39.066445 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tdg6z"] Sep 16 17:46:40.479163 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:46:40.479122 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="effccdf2-43df-4410-b116-c5519137411a" path="/var/lib/kubelet/pods/effccdf2-43df-4410-b116-c5519137411a/volumes" Sep 16 17:48:15.512185 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:15.512151 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerStarted","Data":"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa"} Sep 16 17:48:15.512599 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:15.512205 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerStarted","Data":"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe"} Sep 16 17:48:15.512599 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:15.512381 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:48:15.541428 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:15.541370 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" podStartSLOduration=3.002966349 podStartE2EDuration="2m17.54135389s" totalImagesPullingTime="2m9.144590135s" totalInitContainerRuntime="5.393797406s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:15.540049703 +0000 UTC m=+1403.483371327" watchObservedRunningTime="2026-09-16 17:48:15.54135389 +0000 UTC m=+1403.484675491" Sep 16 17:48:16.517197 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:16.517168 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:48:22.528402 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:48:22.528371 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:49:49.476284 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:49:49.476255 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:49:52.505003 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:49:52.504973 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:49:52.507376 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:49:52.507353 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:50:00.514142 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:00.514090 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 17:50:05.055519 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:05.055485 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:50:05.055984 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:05.055867 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kserve-container" containerID="cri-o://8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe" gracePeriod=30 Sep 16 17:50:05.056055 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:05.055946 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kube-rbac-proxy" containerID="cri-o://f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa" gracePeriod=30 Sep 16 17:50:05.982016 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:05.981978 2536 generic.go:356] "Generic (PLEG): container finished" podID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerID="f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa" exitCode=2 Sep 16 17:50:05.982200 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:05.982055 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerDied","Data":"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa"} Sep 16 17:50:06.497165 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.497144 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:50:06.627464 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627363 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjmkl\" (UniqueName: \"kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl\") pod \"8e09ebd3-183a-4916-9140-1b263ba226f9\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " Sep 16 17:50:06.627464 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627412 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location\") pod \"8e09ebd3-183a-4916-9140-1b263ba226f9\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " Sep 16 17:50:06.627464 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627446 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls\") pod \"8e09ebd3-183a-4916-9140-1b263ba226f9\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " Sep 16 17:50:06.627730 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627485 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert\") pod \"8e09ebd3-183a-4916-9140-1b263ba226f9\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " Sep 16 17:50:06.627730 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627533 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config\") pod \"8e09ebd3-183a-4916-9140-1b263ba226f9\" (UID: \"8e09ebd3-183a-4916-9140-1b263ba226f9\") " Sep 16 17:50:06.627900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627871 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e09ebd3-183a-4916-9140-1b263ba226f9" (UID: "8e09ebd3-183a-4916-9140-1b263ba226f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:50:06.628011 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.627952 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e09ebd3-183a-4916-9140-1b263ba226f9" (UID: "8e09ebd3-183a-4916-9140-1b263ba226f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:50:06.629665 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.629638 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e09ebd3-183a-4916-9140-1b263ba226f9" (UID: "8e09ebd3-183a-4916-9140-1b263ba226f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:50:06.629939 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.629907 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl" (OuterVolumeSpecName: "kube-api-access-tjmkl") pod "8e09ebd3-183a-4916-9140-1b263ba226f9" (UID: "8e09ebd3-183a-4916-9140-1b263ba226f9"). InnerVolumeSpecName "kube-api-access-tjmkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:50:06.632054 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.632033 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e09ebd3-183a-4916-9140-1b263ba226f9" (UID: "8e09ebd3-183a-4916-9140-1b263ba226f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:50:06.728898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.728843 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:50:06.728898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.728892 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:50:06.728898 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.728906 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmkl\" (UniqueName: \"kubernetes.io/projected/8e09ebd3-183a-4916-9140-1b263ba226f9-kube-api-access-tjmkl\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:50:06.729183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.728942 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e09ebd3-183a-4916-9140-1b263ba226f9-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:50:06.729183 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.728955 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e09ebd3-183a-4916-9140-1b263ba226f9-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:50:06.988561 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.988529 2536 generic.go:356] "Generic (PLEG): container finished" podID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerID="8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe" exitCode=0 Sep 16 17:50:06.988747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.988596 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" Sep 16 17:50:06.988747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.988611 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerDied","Data":"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe"} Sep 16 17:50:06.988747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.988669 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml" event={"ID":"8e09ebd3-183a-4916-9140-1b263ba226f9","Type":"ContainerDied","Data":"3d3a5490b9d06449e05599355b20159203fba60439a96f123ce35ee4ace0f331"} Sep 16 17:50:06.988747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.988695 2536 scope.go:117] "RemoveContainer" containerID="f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa" Sep 16 17:50:06.997191 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:06.997172 2536 scope.go:117] "RemoveContainer" containerID="8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe" Sep 16 17:50:07.004071 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.004055 2536 scope.go:117] "RemoveContainer" containerID="720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd" Sep 16 17:50:07.010337 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.010314 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.011505 2536 scope.go:117] "RemoveContainer" containerID="f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:50:07.015281 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa\": container with ID starting with f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa not found: ID does not exist" containerID="f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.015318 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa"} err="failed to get container status \"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa\": rpc error: code = NotFound desc = could not find container \"f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa\": container with ID starting with f5caabbfd27d8596fb98cde84ee1dd02d0064be404a031418d8f49207397bbfa not found: ID does not exist" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.015345 2536 scope.go:117] "RemoveContainer" containerID="8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:50:07.015675 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe\": container with ID starting with 8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe not found: ID does not exist" containerID="8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.015701 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe"} err="failed to get container status \"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe\": rpc error: code = NotFound desc = could not find container \"8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe\": container with ID starting with 8a1f42182b3b5da5f209a6b2c046b8043ed9d7278d617179c6aac77502cd3ebe not found: ID does not exist" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.015723 2536 scope.go:117] "RemoveContainer" containerID="720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:50:07.016034 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd\": container with ID starting with 720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd not found: ID does not exist" containerID="720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.016060 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd"} err="failed to get container status \"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd\": rpc error: code = NotFound desc = could not find container \"720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd\": container with ID starting with 720d291131d04a2c39f39e922abe137fe38b0678c0c4cc0138c8f39d8845f3cd not found: ID does not exist" Sep 16 17:50:07.017043 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:07.016646 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-22dml"] Sep 16 17:50:08.477087 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:08.477055 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" path="/var/lib/kubelet/pods/8e09ebd3-183a-4916-9140-1b263ba226f9/volumes" Sep 16 17:50:10.213328 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213293 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213596 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="storage-initializer" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213607 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="storage-initializer" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213616 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kserve-container" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213621 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kserve-container" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213636 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="storage-initializer" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213642 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="storage-initializer" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213651 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kube-rbac-proxy" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213658 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kube-rbac-proxy" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213665 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213670 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213678 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kube-rbac-proxy" Sep 16 17:50:10.213691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213683 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kube-rbac-proxy" Sep 16 17:50:10.214091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213728 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kserve-container" Sep 16 17:50:10.214091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213735 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kserve-container" Sep 16 17:50:10.214091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213744 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="effccdf2-43df-4410-b116-c5519137411a" containerName="kube-rbac-proxy" Sep 16 17:50:10.214091 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.213750 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e09ebd3-183a-4916-9140-1b263ba226f9" containerName="kube-rbac-proxy" Sep 16 17:50:10.217573 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.217552 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.220589 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220565 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\"" Sep 16 17:50:10.220698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220573 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\"" Sep 16 17:50:10.220698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220646 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:50:10.220698 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220661 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:50:10.220838 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220699 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:50:10.220838 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.220798 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:50:10.221601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.221587 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:50:10.229963 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.229886 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:50:10.361877 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.361842 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.361877 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.361881 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.362136 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.361927 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.362136 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.361956 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.362136 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.362040 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bx5\" (UniqueName: \"kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.462428 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462392 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bx5\" (UniqueName: \"kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.462601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462446 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.462601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462467 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.462601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462499 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.462601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462526 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.463000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.462971 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.463220 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.463202 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.463297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.463280 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.465002 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.464949 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.471635 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.471611 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bx5\" (UniqueName: \"kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.528878 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.528843 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:10.802888 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:10.802861 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:50:10.804865 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:50:10.804839 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76add66e_684d_4986_812f_c8565329afc4.slice/crio-5f4e4e05916ec4cc57ae3d149282eae301a044b4d5cb7cd65f039353f0a170fe WatchSource:0}: Error finding container 5f4e4e05916ec4cc57ae3d149282eae301a044b4d5cb7cd65f039353f0a170fe: Status 404 returned error can't find the container with id 5f4e4e05916ec4cc57ae3d149282eae301a044b4d5cb7cd65f039353f0a170fe Sep 16 17:50:11.009452 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:11.009407 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerStarted","Data":"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f"} Sep 16 17:50:11.009644 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:11.009473 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerStarted","Data":"5f4e4e05916ec4cc57ae3d149282eae301a044b4d5cb7cd65f039353f0a170fe"} Sep 16 17:50:41.137660 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:41.137631 2536 generic.go:356] "Generic (PLEG): container finished" podID="76add66e-684d-4986-812f-c8565329afc4" containerID="2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f" exitCode=0 Sep 16 17:50:41.138085 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:41.137712 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerDied","Data":"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f"} Sep 16 17:50:42.146677 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:42.146639 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerStarted","Data":"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d"} Sep 16 17:50:42.147095 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:42.146731 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerStarted","Data":"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164"} Sep 16 17:50:42.147095 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:42.146958 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:42.147095 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:42.147010 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:50:42.173735 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:42.173678 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" podStartSLOduration=2.576953032 podStartE2EDuration="32.173662599s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.596709567s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:50:42.171393284 +0000 UTC m=+1550.114714959" watchObservedRunningTime="2026-09-16 17:50:42.173662599 +0000 UTC m=+1550.116984200" Sep 16 17:50:48.157638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:50:48.157610 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:52:16.477954 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:16.477905 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:52:26.725496 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:26.725464 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:52:26.725938 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:26.725681 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kserve-container" containerID="cri-o://32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164" gracePeriod=30 Sep 16 17:52:26.725938 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:26.725719 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kube-rbac-proxy" containerID="cri-o://364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d" gracePeriod=30 Sep 16 17:52:27.599521 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:27.599487 2536 generic.go:356] "Generic (PLEG): container finished" podID="76add66e-684d-4986-812f-c8565329afc4" containerID="364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d" exitCode=2 Sep 16 17:52:27.599689 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:27.599534 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerDied","Data":"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d"} Sep 16 17:52:28.152874 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:28.152823 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused" Sep 16 17:52:31.964088 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:31.964063 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:52:32.064282 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064199 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert\") pod \"76add66e-684d-4986-812f-c8565329afc4\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " Sep 16 17:52:32.064282 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064259 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls\") pod \"76add66e-684d-4986-812f-c8565329afc4\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " Sep 16 17:52:32.064473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064304 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95bx5\" (UniqueName: \"kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5\") pod \"76add66e-684d-4986-812f-c8565329afc4\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " Sep 16 17:52:32.064473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064353 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location\") pod \"76add66e-684d-4986-812f-c8565329afc4\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " Sep 16 17:52:32.064473 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064408 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config\") pod \"76add66e-684d-4986-812f-c8565329afc4\" (UID: \"76add66e-684d-4986-812f-c8565329afc4\") " Sep 16 17:52:32.064667 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064627 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76add66e-684d-4986-812f-c8565329afc4" (UID: "76add66e-684d-4986-812f-c8565329afc4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:52:32.064844 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.064822 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76add66e-684d-4986-812f-c8565329afc4" (UID: "76add66e-684d-4986-812f-c8565329afc4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:52:32.066439 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.066414 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76add66e-684d-4986-812f-c8565329afc4" (UID: "76add66e-684d-4986-812f-c8565329afc4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:52:32.066522 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.066508 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5" (OuterVolumeSpecName: "kube-api-access-95bx5") pod "76add66e-684d-4986-812f-c8565329afc4" (UID: "76add66e-684d-4986-812f-c8565329afc4"). InnerVolumeSpecName "kube-api-access-95bx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:52:32.068761 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.068739 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76add66e-684d-4986-812f-c8565329afc4" (UID: "76add66e-684d-4986-812f-c8565329afc4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:52:32.165241 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.165206 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76add66e-684d-4986-812f-c8565329afc4-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:52:32.165241 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.165233 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:52:32.165241 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.165245 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76add66e-684d-4986-812f-c8565329afc4-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:52:32.165466 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.165256 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76add66e-684d-4986-812f-c8565329afc4-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:52:32.165466 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.165263 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95bx5\" (UniqueName: \"kubernetes.io/projected/76add66e-684d-4986-812f-c8565329afc4-kube-api-access-95bx5\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:52:32.621600 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.621566 2536 generic.go:356] "Generic (PLEG): container finished" podID="76add66e-684d-4986-812f-c8565329afc4" containerID="32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164" exitCode=0 Sep 16 17:52:32.621766 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.621610 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerDied","Data":"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164"} Sep 16 17:52:32.621766 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.621672 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" event={"ID":"76add66e-684d-4986-812f-c8565329afc4","Type":"ContainerDied","Data":"5f4e4e05916ec4cc57ae3d149282eae301a044b4d5cb7cd65f039353f0a170fe"} Sep 16 17:52:32.621766 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.621624 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc" Sep 16 17:52:32.621766 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.621709 2536 scope.go:117] "RemoveContainer" containerID="364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d" Sep 16 17:52:32.629792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.629750 2536 scope.go:117] "RemoveContainer" containerID="32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164" Sep 16 17:52:32.636774 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.636755 2536 scope.go:117] "RemoveContainer" containerID="2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f" Sep 16 17:52:32.641587 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.641559 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:52:32.643291 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.643270 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-v59hc"] Sep 16 17:52:32.644744 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.644729 2536 scope.go:117] "RemoveContainer" containerID="364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d" Sep 16 17:52:32.645126 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:52:32.645108 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d\": container with ID starting with 364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d not found: ID does not exist" containerID="364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d" Sep 16 17:52:32.645186 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.645134 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d"} err="failed to get container status \"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d\": rpc error: code = NotFound desc = could not find container \"364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d\": container with ID starting with 364b12f9d5d3f77027fcdb1097cf75cef42203de72310bfdb1242e246137417d not found: ID does not exist" Sep 16 17:52:32.645186 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.645150 2536 scope.go:117] "RemoveContainer" containerID="32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164" Sep 16 17:52:32.645396 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:52:32.645380 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164\": container with ID starting with 32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164 not found: ID does not exist" containerID="32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164" Sep 16 17:52:32.645442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.645401 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164"} err="failed to get container status \"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164\": rpc error: code = NotFound desc = could not find container \"32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164\": container with ID starting with 32b9862723bb39a82d328ea99599640aa2eab598d0e1ec1b3245577541f79164 not found: ID does not exist" Sep 16 17:52:32.645442 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.645418 2536 scope.go:117] "RemoveContainer" containerID="2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f" Sep 16 17:52:32.645629 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:52:32.645615 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f\": container with ID starting with 2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f not found: ID does not exist" containerID="2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f" Sep 16 17:52:32.645665 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:32.645634 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f"} err="failed to get container status \"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f\": rpc error: code = NotFound desc = could not find container \"2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f\": container with ID starting with 2adfd1506d5309c0d4decb9b6015637905bdc849dd35a5a69c7eeba6d0bf760f not found: ID does not exist" Sep 16 17:52:34.477899 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:52:34.477863 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76add66e-684d-4986-812f-c8565329afc4" path="/var/lib/kubelet/pods/76add66e-684d-4986-812f-c8565329afc4/volumes" Sep 16 17:54:33.161531 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.161493 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.161975 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="storage-initializer" Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.161995 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="storage-initializer" Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162007 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kube-rbac-proxy" Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162016 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kube-rbac-proxy" Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162027 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kserve-container" Sep 16 17:54:33.162098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162036 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kserve-container" Sep 16 17:54:33.162410 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162119 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kube-rbac-proxy" Sep 16 17:54:33.162410 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.162128 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="76add66e-684d-4986-812f-c8565329afc4" containerName="kserve-container" Sep 16 17:54:33.165385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.165362 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.174189 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.174167 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:54:33.174324 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.174166 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:54:33.174529 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.174517 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:54:33.175232 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.175216 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:54:33.175450 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.175435 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\"" Sep 16 17:54:33.175516 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.175478 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\"" Sep 16 17:54:33.175717 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.175700 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:54:33.183342 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.183306 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:54:33.261539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.261500 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.261539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.261543 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.261792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.261581 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.261792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.261667 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5t4\" (UniqueName: \"kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.261792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.261706 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.362534 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.362496 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.362718 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.362544 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.362718 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.362606 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.362827 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.362744 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5t4\" (UniqueName: \"kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.362827 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.362779 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.363124 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.363104 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.363332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.363313 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.363404 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.363369 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.364981 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.364963 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.372139 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.372119 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5t4\" (UniqueName: \"kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.476539 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.476506 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:33.753267 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:33.753215 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:54:33.757070 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:54:33.757038 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2df8d1_be78_43cb_85ed_587ee88116a6.slice/crio-c59f8105cc4b603ed27069e2ae00063e61dd058865e58f82619ccd67627ff41d WatchSource:0}: Error finding container c59f8105cc4b603ed27069e2ae00063e61dd058865e58f82619ccd67627ff41d: Status 404 returned error can't find the container with id c59f8105cc4b603ed27069e2ae00063e61dd058865e58f82619ccd67627ff41d Sep 16 17:54:34.136894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:34.136800 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerStarted","Data":"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139"} Sep 16 17:54:34.136894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:34.136880 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerStarted","Data":"c59f8105cc4b603ed27069e2ae00063e61dd058865e58f82619ccd67627ff41d"} Sep 16 17:54:43.174848 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:43.174813 2536 generic.go:356] "Generic (PLEG): container finished" podID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerID="edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139" exitCode=0 Sep 16 17:54:43.175242 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:43.174885 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerDied","Data":"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139"} Sep 16 17:54:44.182781 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:44.182749 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerStarted","Data":"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604"} Sep 16 17:54:44.183278 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:44.182823 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerStarted","Data":"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf"} Sep 16 17:54:44.183278 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:44.183088 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:44.213110 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:44.213055 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podStartSLOduration=2.3171276880000002 podStartE2EDuration="11.213039151s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.895911463s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:54:44.211106899 +0000 UTC m=+1792.154428528" watchObservedRunningTime="2026-09-16 17:54:44.213039151 +0000 UTC m=+1792.156360752" Sep 16 17:54:45.187303 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:45.187215 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:45.188704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:45.188672 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 17:54:46.191613 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:46.191575 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 17:54:51.197017 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:51.196931 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:54:51.197591 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:51.197558 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 17:54:52.531542 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:52.531515 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:54:52.534130 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:54:52.534107 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:55:01.198605 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:01.198572 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:55:19.293853 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:19.293814 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:55:19.294332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:19.294119 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" containerID="cri-o://b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf" gracePeriod=30 Sep 16 17:55:19.294332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:19.294165 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kube-rbac-proxy" containerID="cri-o://4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604" gracePeriod=30 Sep 16 17:55:20.346845 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:20.346813 2536 generic.go:356] "Generic (PLEG): container finished" podID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerID="4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604" exitCode=2 Sep 16 17:55:20.347245 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:20.346888 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerDied","Data":"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604"} Sep 16 17:55:21.140575 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.140550 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:55:21.271233 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271197 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location\") pod \"8f2df8d1-be78-43cb-85ed-587ee88116a6\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " Sep 16 17:55:21.271233 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271243 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert\") pod \"8f2df8d1-be78-43cb-85ed-587ee88116a6\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " Sep 16 17:55:21.271512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271278 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x5t4\" (UniqueName: \"kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4\") pod \"8f2df8d1-be78-43cb-85ed-587ee88116a6\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " Sep 16 17:55:21.271512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271329 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls\") pod \"8f2df8d1-be78-43cb-85ed-587ee88116a6\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " Sep 16 17:55:21.271512 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271371 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config\") pod \"8f2df8d1-be78-43cb-85ed-587ee88116a6\" (UID: \"8f2df8d1-be78-43cb-85ed-587ee88116a6\") " Sep 16 17:55:21.271751 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271706 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f2df8d1-be78-43cb-85ed-587ee88116a6" (UID: "8f2df8d1-be78-43cb-85ed-587ee88116a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:55:21.271884 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.271774 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f2df8d1-be78-43cb-85ed-587ee88116a6" (UID: "8f2df8d1-be78-43cb-85ed-587ee88116a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:55:21.272412 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.272393 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f2df8d1-be78-43cb-85ed-587ee88116a6" (UID: "8f2df8d1-be78-43cb-85ed-587ee88116a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:55:21.273421 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.273395 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f2df8d1-be78-43cb-85ed-587ee88116a6" (UID: "8f2df8d1-be78-43cb-85ed-587ee88116a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:55:21.273540 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.273521 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4" (OuterVolumeSpecName: "kube-api-access-7x5t4") pod "8f2df8d1-be78-43cb-85ed-587ee88116a6" (UID: "8f2df8d1-be78-43cb-85ed-587ee88116a6"). InnerVolumeSpecName "kube-api-access-7x5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:55:21.352796 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.352763 2536 generic.go:356] "Generic (PLEG): container finished" podID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerID="b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf" exitCode=0 Sep 16 17:55:21.353184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.352824 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" Sep 16 17:55:21.353184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.352852 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerDied","Data":"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf"} Sep 16 17:55:21.353184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.352925 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2" event={"ID":"8f2df8d1-be78-43cb-85ed-587ee88116a6","Type":"ContainerDied","Data":"c59f8105cc4b603ed27069e2ae00063e61dd058865e58f82619ccd67627ff41d"} Sep 16 17:55:21.353184 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.352941 2536 scope.go:117] "RemoveContainer" containerID="4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604" Sep 16 17:55:21.361415 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.361396 2536 scope.go:117] "RemoveContainer" containerID="b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf" Sep 16 17:55:21.368671 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.368652 2536 scope.go:117] "RemoveContainer" containerID="edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139" Sep 16 17:55:21.372960 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.372904 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:55:21.372960 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.372958 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2df8d1-be78-43cb-85ed-587ee88116a6-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:55:21.373129 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.372973 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2df8d1-be78-43cb-85ed-587ee88116a6-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:55:21.373129 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.372988 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5t4\" (UniqueName: \"kubernetes.io/projected/8f2df8d1-be78-43cb-85ed-587ee88116a6-kube-api-access-7x5t4\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:55:21.373129 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.373000 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2df8d1-be78-43cb-85ed-587ee88116a6-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:55:21.375571 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.375550 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:55:21.376588 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.376565 2536 scope.go:117] "RemoveContainer" containerID="4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604" Sep 16 17:55:21.376874 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:55:21.376857 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604\": container with ID starting with 4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604 not found: ID does not exist" containerID="4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604" Sep 16 17:55:21.376940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.376882 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604"} err="failed to get container status \"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604\": rpc error: code = NotFound desc = could not find container \"4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604\": container with ID starting with 4bcf04e1563a5f6adb237072793274b1484f18e28fb7203aa3a8d1c6593f5604 not found: ID does not exist" Sep 16 17:55:21.376940 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.376899 2536 scope.go:117] "RemoveContainer" containerID="b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf" Sep 16 17:55:21.377242 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:55:21.377220 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf\": container with ID starting with b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf not found: ID does not exist" containerID="b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf" Sep 16 17:55:21.377319 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.377249 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf"} err="failed to get container status \"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf\": rpc error: code = NotFound desc = could not find container \"b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf\": container with ID starting with b1a5e1c0f2f29147bea3a39926aa23e82ea771f6135e4414163a0b32039a25cf not found: ID does not exist" Sep 16 17:55:21.377319 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.377264 2536 scope.go:117] "RemoveContainer" containerID="edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139" Sep 16 17:55:21.377514 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:55:21.377496 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139\": container with ID starting with edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139 not found: ID does not exist" containerID="edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139" Sep 16 17:55:21.377570 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.377522 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139"} err="failed to get container status \"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139\": rpc error: code = NotFound desc = could not find container \"edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139\": container with ID starting with edc45ad46533f0bf6e7198c8ce9151d9464ec2d85d1704f0edce246c747e6139 not found: ID does not exist" Sep 16 17:55:21.382672 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:21.382646 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sh4w2"] Sep 16 17:55:22.478583 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:22.478551 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" path="/var/lib/kubelet/pods/8f2df8d1-be78-43cb-85ed-587ee88116a6/volumes" Sep 16 17:55:24.420813 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.420781 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421119 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421132 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421143 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="storage-initializer" Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421149 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="storage-initializer" Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421163 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kube-rbac-proxy" Sep 16 17:55:24.421203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421172 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kube-rbac-proxy" Sep 16 17:55:24.421388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421230 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kserve-container" Sep 16 17:55:24.421388 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.421241 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2df8d1-be78-43cb-85ed-587ee88116a6" containerName="kube-rbac-proxy" Sep 16 17:55:24.426116 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.426095 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.428821 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.428794 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\"" Sep 16 17:55:24.429033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.428878 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:55:24.429151 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.429134 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:55:24.429225 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.429205 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:55:24.429311 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.429249 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:55:24.430203 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.430182 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:55:24.430319 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.430236 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\"" Sep 16 17:55:24.440489 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.440465 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:55:24.600335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.600291 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.600529 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.600360 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.600529 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.600397 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm2l9\" (UniqueName: \"kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.600529 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.600433 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.600661 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.600535 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.701626 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701548 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.701626 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701587 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.701820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701717 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.701820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701757 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2l9\" (UniqueName: \"kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.701820 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701795 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.702051 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:55:24.701964 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found Sep 16 17:55:24.702051 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.701966 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.702051 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:55:24.702035 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls podName:9e729b49-ab77-4774-a01b-4d535a13d54d nodeName:}" failed. No retries permitted until 2026-09-16 17:55:25.202014612 +0000 UTC m=+1833.145336202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found Sep 16 17:55:24.702306 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.702278 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.702419 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.702399 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:24.713276 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:24.713257 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2l9\" (UniqueName: \"kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:25.206747 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:25.206710 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:25.209125 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:25.209100 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:25.337228 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:25.337187 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:25.613965 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:25.613939 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:55:25.616669 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:55:25.616640 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e729b49_ab77_4774_a01b_4d535a13d54d.slice/crio-9ff48bf964dfa5c67d56afa4e79406752c35c733ed5aa9ea622731500c0c3851 WatchSource:0}: Error finding container 9ff48bf964dfa5c67d56afa4e79406752c35c733ed5aa9ea622731500c0c3851: Status 404 returned error can't find the container with id 9ff48bf964dfa5c67d56afa4e79406752c35c733ed5aa9ea622731500c0c3851 Sep 16 17:55:26.376870 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:26.376834 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerStarted","Data":"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192"} Sep 16 17:55:26.377070 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:26.376897 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerStarted","Data":"9ff48bf964dfa5c67d56afa4e79406752c35c733ed5aa9ea622731500c0c3851"} Sep 16 17:55:30.395445 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:30.395411 2536 generic.go:356] "Generic (PLEG): container finished" podID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerID="787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192" exitCode=0 Sep 16 17:55:30.395849 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:30.395471 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerDied","Data":"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192"} Sep 16 17:55:31.402460 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:31.402429 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerStarted","Data":"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0"} Sep 16 17:55:31.402861 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:31.402488 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerStarted","Data":"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac"} Sep 16 17:55:31.402861 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:31.402668 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:32.408116 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:32.408087 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:32.409465 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:32.409433 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 17:55:33.412391 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:33.412351 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 17:55:38.416691 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:38.416660 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:38.417163 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:38.417119 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 17:55:38.441805 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:38.441740 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podStartSLOduration=9.809050942 podStartE2EDuration="14.441719925s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.632668983s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:31.429082087 +0000 UTC m=+1839.372403688" watchObservedRunningTime="2026-09-16 17:55:38.441719925 +0000 UTC m=+1846.385041528" Sep 16 17:55:48.417969 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:48.417939 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:55:59.554706 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:59.554662 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:55:59.555223 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:59.555026 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" containerID="cri-o://0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac" gracePeriod=30 Sep 16 17:55:59.555223 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:55:59.555058 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kube-rbac-proxy" containerID="cri-o://50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0" gracePeriod=30 Sep 16 17:56:00.533680 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:00.533646 2536 generic.go:356] "Generic (PLEG): container finished" podID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerID="50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0" exitCode=2 Sep 16 17:56:00.533845 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:00.533690 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerDied","Data":"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0"} Sep 16 17:56:00.997923 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:00.997891 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:56:01.106380 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106293 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location\") pod \"9e729b49-ab77-4774-a01b-4d535a13d54d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " Sep 16 17:56:01.106380 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106353 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert\") pod \"9e729b49-ab77-4774-a01b-4d535a13d54d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " Sep 16 17:56:01.106566 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106384 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") pod \"9e729b49-ab77-4774-a01b-4d535a13d54d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " Sep 16 17:56:01.106566 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106413 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm2l9\" (UniqueName: \"kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9\") pod \"9e729b49-ab77-4774-a01b-4d535a13d54d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " Sep 16 17:56:01.106566 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106439 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config\") pod \"9e729b49-ab77-4774-a01b-4d535a13d54d\" (UID: \"9e729b49-ab77-4774-a01b-4d535a13d54d\") " Sep 16 17:56:01.106804 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106776 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e729b49-ab77-4774-a01b-4d535a13d54d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:56:01.106872 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.106837 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e729b49-ab77-4774-a01b-4d535a13d54d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:56:01.107309 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.107290 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e729b49-ab77-4774-a01b-4d535a13d54d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:56:01.108509 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.108487 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9" (OuterVolumeSpecName: "kube-api-access-gm2l9") pod "9e729b49-ab77-4774-a01b-4d535a13d54d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d"). InnerVolumeSpecName "kube-api-access-gm2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:56:01.108579 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.108506 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e729b49-ab77-4774-a01b-4d535a13d54d" (UID: "9e729b49-ab77-4774-a01b-4d535a13d54d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:56:01.207937 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.207880 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e729b49-ab77-4774-a01b-4d535a13d54d-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:56:01.207937 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.207935 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm2l9\" (UniqueName: \"kubernetes.io/projected/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-api-access-gm2l9\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:56:01.208132 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.207951 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:56:01.208132 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.207960 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e729b49-ab77-4774-a01b-4d535a13d54d-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:56:01.208132 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.207971 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e729b49-ab77-4774-a01b-4d535a13d54d-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:56:01.539330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.539292 2536 generic.go:356] "Generic (PLEG): container finished" podID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerID="0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac" exitCode=0 Sep 16 17:56:01.539532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.539346 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" Sep 16 17:56:01.539532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.539386 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerDied","Data":"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac"} Sep 16 17:56:01.539532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.539438 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d" event={"ID":"9e729b49-ab77-4774-a01b-4d535a13d54d","Type":"ContainerDied","Data":"9ff48bf964dfa5c67d56afa4e79406752c35c733ed5aa9ea622731500c0c3851"} Sep 16 17:56:01.539532 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.539464 2536 scope.go:117] "RemoveContainer" containerID="50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0" Sep 16 17:56:01.548127 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.548111 2536 scope.go:117] "RemoveContainer" containerID="0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac" Sep 16 17:56:01.555343 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.555325 2536 scope.go:117] "RemoveContainer" containerID="787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192" Sep 16 17:56:01.562281 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.562261 2536 scope.go:117] "RemoveContainer" containerID="50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0" Sep 16 17:56:01.562540 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:56:01.562520 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0\": container with ID starting with 50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0 not found: ID does not exist" containerID="50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0" Sep 16 17:56:01.562601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.562547 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0"} err="failed to get container status \"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0\": rpc error: code = NotFound desc = could not find container \"50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0\": container with ID starting with 50a9c13fc88a4dfc0054f05f3005906266b4cbbda11c8614cd84ed324318e5f0 not found: ID does not exist" Sep 16 17:56:01.562601 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.562565 2536 scope.go:117] "RemoveContainer" containerID="0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac" Sep 16 17:56:01.562824 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:56:01.562807 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac\": container with ID starting with 0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac not found: ID does not exist" containerID="0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac" Sep 16 17:56:01.562869 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.562830 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac"} err="failed to get container status \"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac\": rpc error: code = NotFound desc = could not find container \"0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac\": container with ID starting with 0599eba7d4ba5e06d7c074bf1e5382332bebdb241b35f7a90fe02fcbb4f1ffac not found: ID does not exist" Sep 16 17:56:01.562869 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.562845 2536 scope.go:117] "RemoveContainer" containerID="787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192" Sep 16 17:56:01.563122 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:56:01.563106 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192\": container with ID starting with 787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192 not found: ID does not exist" containerID="787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192" Sep 16 17:56:01.563182 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.563125 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192"} err="failed to get container status \"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192\": rpc error: code = NotFound desc = could not find container \"787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192\": container with ID starting with 787186c4396d6d481dd68808da78edb009be4ccd2cfe4897901e51a691b99192 not found: ID does not exist" Sep 16 17:56:01.565809 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.565786 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:56:01.572174 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:01.572153 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4b52d"] Sep 16 17:56:02.477749 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:02.477715 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" path="/var/lib/kubelet/pods/9e729b49-ab77-4774-a01b-4d535a13d54d/volumes" Sep 16 17:56:05.019715 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.019681 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020024 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="storage-initializer" Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020037 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="storage-initializer" Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020050 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020055 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020068 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kube-rbac-proxy" Sep 16 17:56:05.020104 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020073 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kube-rbac-proxy" Sep 16 17:56:05.020288 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020125 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kube-rbac-proxy" Sep 16 17:56:05.020288 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.020135 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e729b49-ab77-4774-a01b-4d535a13d54d" containerName="kserve-container" Sep 16 17:56:05.024846 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.024827 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.030334 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.030314 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:56:05.030557 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.030536 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Sep 16 17:56:05.030682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.030588 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:56:05.030682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.030665 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Sep 16 17:56:05.031452 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.031436 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:56:05.031862 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.031849 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:56:05.033077 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.033060 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:56:05.042894 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.042870 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:56:05.137420 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.137382 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.137420 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.137421 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.137618 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.137542 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.137618 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.137584 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.137690 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.137636 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtr6x\" (UniqueName: \"kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238362 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238328 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238530 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238391 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238530 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238419 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238530 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238444 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtr6x\" (UniqueName: \"kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238530 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238467 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.238530 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:56:05.238504 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found Sep 16 17:56:05.238784 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:56:05.238578 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls podName:58fd598b-6283-422b-bb55-7578011b5e62 nodeName:}" failed. No retries permitted until 2026-09-16 17:56:05.738556792 +0000 UTC m=+1873.681878372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls") pod "isvc-canary-create-predictor-854596548c-wg76h" (UID: "58fd598b-6283-422b-bb55-7578011b5e62") : secret "isvc-canary-create-predictor-serving-cert" not found Sep 16 17:56:05.238888 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.238869 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.239237 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.239212 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.239339 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.239234 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.253787 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.253765 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtr6x\" (UniqueName: \"kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.743000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.742958 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.745327 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.745303 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-wg76h\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:05.935333 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:05.935275 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:06.217595 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:06.217507 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:56:06.221298 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:56:06.221271 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fd598b_6283_422b_bb55_7578011b5e62.slice/crio-10bafcfc103682f9360189bb4211710b8aa5e93dc784e0c06d33573f2b7a69a0 WatchSource:0}: Error finding container 10bafcfc103682f9360189bb4211710b8aa5e93dc784e0c06d33573f2b7a69a0: Status 404 returned error can't find the container with id 10bafcfc103682f9360189bb4211710b8aa5e93dc784e0c06d33573f2b7a69a0 Sep 16 17:56:06.564819 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:06.564786 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerStarted","Data":"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f"} Sep 16 17:56:06.564993 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:06.564845 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerStarted","Data":"10bafcfc103682f9360189bb4211710b8aa5e93dc784e0c06d33573f2b7a69a0"} Sep 16 17:56:10.583887 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:10.583850 2536 generic.go:356] "Generic (PLEG): container finished" podID="58fd598b-6283-422b-bb55-7578011b5e62" containerID="baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f" exitCode=0 Sep 16 17:56:10.584273 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:10.583892 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerDied","Data":"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f"} Sep 16 17:56:10.585077 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:10.585053 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 17:56:24.661576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:24.661540 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerStarted","Data":"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d"} Sep 16 17:56:24.662064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:24.661602 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerStarted","Data":"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd"} Sep 16 17:56:24.662064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:24.661830 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:24.662064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:24.662004 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:24.663383 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:24.663356 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:56:25.665720 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:25.665674 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:56:30.670726 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:30.670696 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:56:30.671216 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:30.671190 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:56:30.703090 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:30.703039 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podStartSLOduration=9.097734472 podStartE2EDuration="26.703023575s" totalImagesPullingTime="13.507672025s" totalInitContainerRuntime="4.097617078s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:24.709680225 +0000 UTC m=+1892.653001829" watchObservedRunningTime="2026-09-16 17:56:30.703023575 +0000 UTC m=+1898.646345180" Sep 16 17:56:40.671407 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:40.671359 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:56:50.671565 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:56:50.671525 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:57:00.672113 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:00.672071 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:57:10.671553 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:10.671514 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:57:20.671674 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:20.671633 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:57:30.671733 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:30.671691 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 17:57:40.672166 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:40.672134 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:57:44.960201 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:44.960121 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:57:44.960638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:44.960375 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" containerID="cri-o://47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd" gracePeriod=30 Sep 16 17:57:44.960638 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:44.960500 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kube-rbac-proxy" containerID="cri-o://5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d" gracePeriod=30 Sep 16 17:57:45.666428 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:45.666385 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused" Sep 16 17:57:46.021791 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:46.021761 2536 generic.go:356] "Generic (PLEG): container finished" podID="58fd598b-6283-422b-bb55-7578011b5e62" containerID="5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d" exitCode=2 Sep 16 17:57:46.022182 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:46.021796 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerDied","Data":"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d"} Sep 16 17:57:49.802316 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.802291 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:57:49.927434 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927355 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location\") pod \"58fd598b-6283-422b-bb55-7578011b5e62\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " Sep 16 17:57:49.927434 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927389 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert\") pod \"58fd598b-6283-422b-bb55-7578011b5e62\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " Sep 16 17:57:49.927434 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927428 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") pod \"58fd598b-6283-422b-bb55-7578011b5e62\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " Sep 16 17:57:49.927704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927457 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtr6x\" (UniqueName: \"kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x\") pod \"58fd598b-6283-422b-bb55-7578011b5e62\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " Sep 16 17:57:49.927704 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927517 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config\") pod \"58fd598b-6283-422b-bb55-7578011b5e62\" (UID: \"58fd598b-6283-422b-bb55-7578011b5e62\") " Sep 16 17:57:49.927814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927721 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58fd598b-6283-422b-bb55-7578011b5e62" (UID: "58fd598b-6283-422b-bb55-7578011b5e62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:57:49.927814 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.927745 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58fd598b-6283-422b-bb55-7578011b5e62" (UID: "58fd598b-6283-422b-bb55-7578011b5e62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:57:49.928072 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.928041 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58fd598b-6283-422b-bb55-7578011b5e62" (UID: "58fd598b-6283-422b-bb55-7578011b5e62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:57:49.929639 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.929619 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58fd598b-6283-422b-bb55-7578011b5e62" (UID: "58fd598b-6283-422b-bb55-7578011b5e62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:57:49.929790 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:49.929770 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x" (OuterVolumeSpecName: "kube-api-access-jtr6x") pod "58fd598b-6283-422b-bb55-7578011b5e62" (UID: "58fd598b-6283-422b-bb55-7578011b5e62"). InnerVolumeSpecName "kube-api-access-jtr6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:57:50.029033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.029000 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:57:50.029033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.029029 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fd598b-6283-422b-bb55-7578011b5e62-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:57:50.029033 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.029039 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fd598b-6283-422b-bb55-7578011b5e62-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:57:50.029270 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.029050 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58fd598b-6283-422b-bb55-7578011b5e62-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:57:50.029270 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.029058 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtr6x\" (UniqueName: \"kubernetes.io/projected/58fd598b-6283-422b-bb55-7578011b5e62-kube-api-access-jtr6x\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:57:50.041688 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.041647 2536 generic.go:356] "Generic (PLEG): container finished" podID="58fd598b-6283-422b-bb55-7578011b5e62" containerID="47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd" exitCode=0 Sep 16 17:57:50.041847 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.041717 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" Sep 16 17:57:50.041847 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.041739 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerDied","Data":"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd"} Sep 16 17:57:50.041847 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.041813 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h" event={"ID":"58fd598b-6283-422b-bb55-7578011b5e62","Type":"ContainerDied","Data":"10bafcfc103682f9360189bb4211710b8aa5e93dc784e0c06d33573f2b7a69a0"} Sep 16 17:57:50.042014 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.041857 2536 scope.go:117] "RemoveContainer" containerID="5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d" Sep 16 17:57:50.051361 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.051295 2536 scope.go:117] "RemoveContainer" containerID="47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd" Sep 16 17:57:50.059432 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.059410 2536 scope.go:117] "RemoveContainer" containerID="baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f" Sep 16 17:57:50.067451 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.067428 2536 scope.go:117] "RemoveContainer" containerID="5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d" Sep 16 17:57:50.067810 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:57:50.067785 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d\": container with ID starting with 5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d not found: ID does not exist" containerID="5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d" Sep 16 17:57:50.067881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.067822 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d"} err="failed to get container status \"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d\": rpc error: code = NotFound desc = could not find container \"5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d\": container with ID starting with 5df9efeb84649e2d49b794d03936d36f08eb179fe5a1d7a8165ec56538422b1d not found: ID does not exist" Sep 16 17:57:50.067881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.067840 2536 scope.go:117] "RemoveContainer" containerID="47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd" Sep 16 17:57:50.068147 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:57:50.068130 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd\": container with ID starting with 47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd not found: ID does not exist" containerID="47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd" Sep 16 17:57:50.068190 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.068154 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd"} err="failed to get container status \"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd\": rpc error: code = NotFound desc = could not find container \"47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd\": container with ID starting with 47d3eb9f5d8d9b6bf56a1954499ab98b0f188ad6574a63eb36e167c0c8b67ccd not found: ID does not exist" Sep 16 17:57:50.068190 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.068170 2536 scope.go:117] "RemoveContainer" containerID="baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f" Sep 16 17:57:50.068439 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:57:50.068418 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f\": container with ID starting with baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f not found: ID does not exist" containerID="baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f" Sep 16 17:57:50.068499 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.068447 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f"} err="failed to get container status \"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f\": rpc error: code = NotFound desc = could not find container \"baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f\": container with ID starting with baa56c59ca74f8e9c23ce6b95e8d3b0493395ac81a1dfa5c63a34a280356648f not found: ID does not exist" Sep 16 17:57:50.077932 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.077890 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:57:50.089701 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.089679 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-wg76h"] Sep 16 17:57:50.119844 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.119809 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120201 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kube-rbac-proxy" Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120223 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kube-rbac-proxy" Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120258 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="storage-initializer" Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120267 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="storage-initializer" Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120275 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" Sep 16 17:57:50.120280 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120284 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" Sep 16 17:57:50.120585 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120363 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kube-rbac-proxy" Sep 16 17:57:50.120585 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.120379 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="58fd598b-6283-422b-bb55-7578011b5e62" containerName="kserve-container" Sep 16 17:57:50.125189 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.125171 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.130103 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.130078 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Sep 16 17:57:50.133536 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.133517 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Sep 16 17:57:50.134900 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.134885 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 17:57:50.135614 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.135596 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 17:57:50.138220 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.138202 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 17:57:50.138306 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.138247 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 17:57:50.138306 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.138253 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 17:57:50.141406 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.141386 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 17:57:50.171395 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.171361 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:57:50.174752 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.174730 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.177905 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.177858 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Sep 16 17:57:50.192373 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.192350 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:57:50.231596 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231562 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvvn\" (UniqueName: \"kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.231596 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231593 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231615 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231660 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxb4\" (UniqueName: \"kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231700 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231743 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231765 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231794 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.231808 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231808 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.232081 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.231827 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333078 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333021 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333078 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333077 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333111 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333127 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333146 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333210 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvvn\" (UniqueName: \"kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333233 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333283 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.333330 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333315 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxb4\" (UniqueName: \"kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.333675 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333344 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.333675 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:57:50.333485 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found Sep 16 17:57:50.333675 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333517 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.333675 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:57:50.333543 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls podName:707d23cd-0a28-47da-834e-4390f5672686 nodeName:}" failed. No retries permitted until 2026-09-16 17:57:50.833523627 +0000 UTC m=+1978.776845210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls") pod "isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" (UID: "707d23cd-0a28-47da-834e-4390f5672686") : secret "isvc-canary-promote-predictor-serving-cert" not found Sep 16 17:57:50.333901 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333859 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.334000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333909 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.334000 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.333956 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.334098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.334016 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.334326 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.334305 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.336166 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.336145 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.345086 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.345060 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxb4\" (UniqueName: \"kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.346663 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.346644 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvvn\" (UniqueName: \"kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-45dtx\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.434501 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.434422 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:50.479583 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.479544 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58fd598b-6283-422b-bb55-7578011b5e62" path="/var/lib/kubelet/pods/58fd598b-6283-422b-bb55-7578011b5e62/volumes" Sep 16 17:57:50.711633 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.711609 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 17:57:50.713609 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:57:50.713583 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269db67_420d_426a_b8f6_adb240c8a6a6.slice/crio-a5e2cc1ee6d812bd70459455e8f8d8329dc963f535f809a71104231c9a28cf47 WatchSource:0}: Error finding container a5e2cc1ee6d812bd70459455e8f8d8329dc963f535f809a71104231c9a28cf47: Status 404 returned error can't find the container with id a5e2cc1ee6d812bd70459455e8f8d8329dc963f535f809a71104231c9a28cf47 Sep 16 17:57:50.837673 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.837631 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:50.839971 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:50.839951 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-2s5d2\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:51.048414 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:51.048379 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerStarted","Data":"817e31b11c73c1596676b4a5890c91242d615c3e37ea0b4789ead28c7082e00e"} Sep 16 17:57:51.048635 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:51.048449 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerStarted","Data":"a5e2cc1ee6d812bd70459455e8f8d8329dc963f535f809a71104231c9a28cf47"} Sep 16 17:57:51.084878 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:51.084840 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:51.366229 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:51.366152 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:57:51.369656 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:57:51.369631 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707d23cd_0a28_47da_834e_4390f5672686.slice/crio-223505018bba7f7236a10db2c95d8c007bad85a1f14747ac8371369f8c6017c9 WatchSource:0}: Error finding container 223505018bba7f7236a10db2c95d8c007bad85a1f14747ac8371369f8c6017c9: Status 404 returned error can't find the container with id 223505018bba7f7236a10db2c95d8c007bad85a1f14747ac8371369f8c6017c9 Sep 16 17:57:52.055319 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:52.055285 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerStarted","Data":"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a"} Sep 16 17:57:52.055792 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:52.055344 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerStarted","Data":"223505018bba7f7236a10db2c95d8c007bad85a1f14747ac8371369f8c6017c9"} Sep 16 17:57:55.069399 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:55.069366 2536 generic.go:356] "Generic (PLEG): container finished" podID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerID="817e31b11c73c1596676b4a5890c91242d615c3e37ea0b4789ead28c7082e00e" exitCode=0 Sep 16 17:57:55.069802 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:55.069405 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerDied","Data":"817e31b11c73c1596676b4a5890c91242d615c3e37ea0b4789ead28c7082e00e"} Sep 16 17:57:56.076244 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.076207 2536 generic.go:356] "Generic (PLEG): container finished" podID="707d23cd-0a28-47da-834e-4390f5672686" containerID="d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a" exitCode=0 Sep 16 17:57:56.076682 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.076281 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerDied","Data":"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a"} Sep 16 17:57:56.078606 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.078587 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerStarted","Data":"1a590c5e967d8bb5fcba50fa396b701c4591d9e975060e41aa787d676a44ce4d"} Sep 16 17:57:56.078695 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.078634 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerStarted","Data":"62ab54f2eee57c1ccf22438f90dda7d9cd75b0dbe4654b063c513dc0955cef2b"} Sep 16 17:57:56.078826 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.078809 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:56.078875 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.078852 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:57:56.080470 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.080441 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:57:56.125075 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:56.125031 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podStartSLOduration=2.238979222 podStartE2EDuration="6.125013952s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88603473s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:57:56.124777117 +0000 UTC m=+1984.068098720" watchObservedRunningTime="2026-09-16 17:57:56.125013952 +0000 UTC m=+1984.068335554" Sep 16 17:57:57.085141 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.085110 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerStarted","Data":"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1"} Sep 16 17:57:57.085554 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.085174 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerStarted","Data":"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af"} Sep 16 17:57:57.085554 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.085376 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:57:57.085645 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.085591 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:57.085687 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.085652 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:57:57.087096 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:57.087072 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:57:58.090650 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:57:58.090611 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:02.090323 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:02.090294 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:58:02.091101 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:02.091066 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:02.111578 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:02.111526 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podStartSLOduration=8.437011694 podStartE2EDuration="12.111510532s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.674498838s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:57:57.115119059 +0000 UTC m=+1985.058440683" watchObservedRunningTime="2026-09-16 17:58:02.111510532 +0000 UTC m=+1990.054832133" Sep 16 17:58:03.097472 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:03.097436 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:58:03.098202 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:03.098172 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:12.090976 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:12.090935 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:13.099089 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:13.099042 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:22.091662 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:22.091612 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:23.098426 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:23.098389 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:32.091516 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:32.091469 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:33.098482 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:33.098444 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:42.091309 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:42.091261 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:43.098233 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:43.098189 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:58:52.091229 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:52.091184 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:58:53.098214 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:58:53.098176 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:59:02.091062 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:02.091015 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:59:03.099073 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:03.099036 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:59:03.472861 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:03.472823 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 17:59:04.473139 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:04.473093 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 17:59:13.473092 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:13.473058 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 17:59:13.629057 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:13.629016 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:59:13.629335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:13.629302 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" containerID="cri-o://dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af" gracePeriod=30 Sep 16 17:59:13.629335 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:13.629320 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kube-rbac-proxy" containerID="cri-o://72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1" gracePeriod=30 Sep 16 17:59:13.630098 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:13.630078 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:59:14.428828 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:14.428797 2536 generic.go:356] "Generic (PLEG): container finished" podID="707d23cd-0a28-47da-834e-4390f5672686" containerID="72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1" exitCode=2 Sep 16 17:59:14.429026 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:14.428875 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerDied","Data":"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1"} Sep 16 17:59:18.091809 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.091721 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused" Sep 16 17:59:18.375876 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.375852 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:59:18.448064 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.448032 2536 generic.go:356] "Generic (PLEG): container finished" podID="707d23cd-0a28-47da-834e-4390f5672686" containerID="dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af" exitCode=0 Sep 16 17:59:18.448221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.448077 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerDied","Data":"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af"} Sep 16 17:59:18.448221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.448086 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" Sep 16 17:59:18.448221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.448150 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2" event={"ID":"707d23cd-0a28-47da-834e-4390f5672686","Type":"ContainerDied","Data":"223505018bba7f7236a10db2c95d8c007bad85a1f14747ac8371369f8c6017c9"} Sep 16 17:59:18.448221 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.448176 2536 scope.go:117] "RemoveContainer" containerID="72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1" Sep 16 17:59:18.456013 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.455979 2536 scope.go:117] "RemoveContainer" containerID="dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af" Sep 16 17:59:18.463247 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.463221 2536 scope.go:117] "RemoveContainer" containerID="d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a" Sep 16 17:59:18.470309 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.470286 2536 scope.go:117] "RemoveContainer" containerID="72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1" Sep 16 17:59:18.470584 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:59:18.470565 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1\": container with ID starting with 72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1 not found: ID does not exist" containerID="72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1" Sep 16 17:59:18.470636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.470595 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1"} err="failed to get container status \"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1\": rpc error: code = NotFound desc = could not find container \"72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1\": container with ID starting with 72b3c3c780f9c2acdb26f3cdf2eef3e2201c5e7c359e2d6e578e96ada1dda9a1 not found: ID does not exist" Sep 16 17:59:18.470636 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.470612 2536 scope.go:117] "RemoveContainer" containerID="dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af" Sep 16 17:59:18.470877 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:59:18.470858 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af\": container with ID starting with dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af not found: ID does not exist" containerID="dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af" Sep 16 17:59:18.470952 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.470883 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af"} err="failed to get container status \"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af\": rpc error: code = NotFound desc = could not find container \"dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af\": container with ID starting with dd5a9bc0c72851f5f0d7a0c2fdc79ae80c16e5a2b3f2209429eca84f7ccc84af not found: ID does not exist" Sep 16 17:59:18.470952 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.470902 2536 scope.go:117] "RemoveContainer" containerID="d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a" Sep 16 17:59:18.471136 ip-10-0-130-222 kubenswrapper[2536]: E0916 17:59:18.471122 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a\": container with ID starting with d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a not found: ID does not exist" containerID="d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a" Sep 16 17:59:18.471176 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.471138 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a"} err="failed to get container status \"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a\": rpc error: code = NotFound desc = could not find container \"d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a\": container with ID starting with d05c6494565872882624842b8e79c74fe2b7b9803ff2718b1aa797b2750f865a not found: ID does not exist" Sep 16 17:59:18.484995 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.484967 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxb4\" (UniqueName: \"kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4\") pod \"707d23cd-0a28-47da-834e-4390f5672686\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " Sep 16 17:59:18.485115 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485001 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") pod \"707d23cd-0a28-47da-834e-4390f5672686\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " Sep 16 17:59:18.485115 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485025 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert\") pod \"707d23cd-0a28-47da-834e-4390f5672686\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " Sep 16 17:59:18.485115 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485064 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config\") pod \"707d23cd-0a28-47da-834e-4390f5672686\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " Sep 16 17:59:18.485115 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485113 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location\") pod \"707d23cd-0a28-47da-834e-4390f5672686\" (UID: \"707d23cd-0a28-47da-834e-4390f5672686\") " Sep 16 17:59:18.485456 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485426 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "707d23cd-0a28-47da-834e-4390f5672686" (UID: "707d23cd-0a28-47da-834e-4390f5672686"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:59:18.485531 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485441 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "707d23cd-0a28-47da-834e-4390f5672686" (UID: "707d23cd-0a28-47da-834e-4390f5672686"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 17:59:18.485531 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.485495 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "707d23cd-0a28-47da-834e-4390f5672686" (UID: "707d23cd-0a28-47da-834e-4390f5672686"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 17:59:18.486988 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.486967 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "707d23cd-0a28-47da-834e-4390f5672686" (UID: "707d23cd-0a28-47da-834e-4390f5672686"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 17:59:18.487137 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.487117 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4" (OuterVolumeSpecName: "kube-api-access-tjxb4") pod "707d23cd-0a28-47da-834e-4390f5672686" (UID: "707d23cd-0a28-47da-834e-4390f5672686"). InnerVolumeSpecName "kube-api-access-tjxb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 17:59:18.586332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.586294 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:59:18.586332 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.586321 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707d23cd-0a28-47da-834e-4390f5672686-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:59:18.586538 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.586350 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxb4\" (UniqueName: \"kubernetes.io/projected/707d23cd-0a28-47da-834e-4390f5672686-kube-api-access-tjxb4\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:59:18.586538 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.586361 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707d23cd-0a28-47da-834e-4390f5672686-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:59:18.586538 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.586369 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707d23cd-0a28-47da-834e-4390f5672686-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 17:59:18.773875 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.773835 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:59:18.778976 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:18.778950 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-2s5d2"] Sep 16 17:59:20.478775 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:20.478744 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707d23cd-0a28-47da-834e-4390f5672686" path="/var/lib/kubelet/pods/707d23cd-0a28-47da-834e-4390f5672686/volumes" Sep 16 17:59:30.118006 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.117973 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118331 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kube-rbac-proxy" Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118343 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kube-rbac-proxy" Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118359 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="storage-initializer" Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118365 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="storage-initializer" Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118379 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" Sep 16 17:59:30.118385 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118385 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" Sep 16 17:59:30.118576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118443 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kube-rbac-proxy" Sep 16 17:59:30.118576 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.118452 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="707d23cd-0a28-47da-834e-4390f5672686" containerName="kserve-container" Sep 16 17:59:30.121415 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.121399 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.125548 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.125515 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\"" Sep 16 17:59:30.125676 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.125517 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\"" Sep 16 17:59:30.145805 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.145773 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 17:59:30.177568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.177533 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.177568 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.177570 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.177764 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.177602 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.177764 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.177683 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.177764 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.177720 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktngk\" (UniqueName: \"kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279032 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.278993 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279040 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktngk\" (UniqueName: \"kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279227 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279119 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279357 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279310 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279357 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279350 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279537 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279518 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279798 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279779 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.279896 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.279875 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.281776 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.281759 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.291841 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.291809 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktngk\" (UniqueName: \"kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk\") pod \"isvc-triton-predictor-5d7db58676-pq2fx\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.431756 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.431667 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 17:59:30.714648 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:30.714623 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 17:59:30.716687 ip-10-0-130-222 kubenswrapper[2536]: W0916 17:59:30.716658 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d3c2f6_2c36_4147_8bf9_97786486a3b3.slice/crio-feaa1e14566f618e1841e6b94303d3cee90e4c75daf6af9efcc662c05c615bce WatchSource:0}: Error finding container feaa1e14566f618e1841e6b94303d3cee90e4c75daf6af9efcc662c05c615bce: Status 404 returned error can't find the container with id feaa1e14566f618e1841e6b94303d3cee90e4c75daf6af9efcc662c05c615bce Sep 16 17:59:31.508870 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:31.508831 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerStarted","Data":"ce1f165f7096990047bcb0962e88763d34a8f2841091fe56171846ac33844806"} Sep 16 17:59:31.509344 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:31.508889 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerStarted","Data":"feaa1e14566f618e1841e6b94303d3cee90e4c75daf6af9efcc662c05c615bce"} Sep 16 17:59:34.522947 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:34.522899 2536 generic.go:356] "Generic (PLEG): container finished" podID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerID="ce1f165f7096990047bcb0962e88763d34a8f2841091fe56171846ac33844806" exitCode=0 Sep 16 17:59:34.523297 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:34.522973 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerDied","Data":"ce1f165f7096990047bcb0962e88763d34a8f2841091fe56171846ac33844806"} Sep 16 17:59:52.574881 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:52.574852 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 17:59:52.579034 ip-10-0-130-222 kubenswrapper[2536]: I0916 17:59:52.579005 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:00:40.349409 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:40.349375 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 18:00:40.349976 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:40.349656 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" containerID="cri-o://62ab54f2eee57c1ccf22438f90dda7d9cd75b0dbe4654b063c513dc0955cef2b" gracePeriod=30 Sep 16 18:00:40.349976 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:40.349709 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" containerID="cri-o://1a590c5e967d8bb5fcba50fa396b701c4591d9e975060e41aa787d676a44ce4d" gracePeriod=30 Sep 16 18:00:42.086170 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:42.086119 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused" Sep 16 18:00:43.473011 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:43.472965 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 18:00:45.701244 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.701199 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:00:45.724942 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.724892 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:00:45.725129 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.725037 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:45.727862 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.727775 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 16 18:00:45.727862 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.727810 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 16 18:00:45.911766 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.911727 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:45.912066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.911794 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgtk\" (UniqueName: \"kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:45.912066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.911893 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:45.912066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.911999 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:45.912066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:45.912042 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.012516 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.012476 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.012808 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.012549 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgtk\" (UniqueName: \"kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.012808 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.012590 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.012808 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.012721 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.012808 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.012765 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.013376 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.013307 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.013478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.013440 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.013540 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.013501 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.015379 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.015359 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.021806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.021780 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgtk\" (UniqueName: \"kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-frhkp\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.096425 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.096376 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:00:46.966062 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.966018 2536 generic.go:356] "Generic (PLEG): container finished" podID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerID="1a590c5e967d8bb5fcba50fa396b701c4591d9e975060e41aa787d676a44ce4d" exitCode=2 Sep 16 18:00:46.966062 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.966056 2536 generic.go:356] "Generic (PLEG): container finished" podID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerID="62ab54f2eee57c1ccf22438f90dda7d9cd75b0dbe4654b063c513dc0955cef2b" exitCode=0 Sep 16 18:00:46.966491 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.966082 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerDied","Data":"1a590c5e967d8bb5fcba50fa396b701c4591d9e975060e41aa787d676a44ce4d"} Sep 16 18:00:46.966491 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:46.966169 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerDied","Data":"62ab54f2eee57c1ccf22438f90dda7d9cd75b0dbe4654b063c513dc0955cef2b"} Sep 16 18:00:47.155704 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.155675 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 18:00:47.223934 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.223883 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qvvn\" (UniqueName: \"kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn\") pod \"8269db67-420d-426a-b8f6-adb240c8a6a6\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " Sep 16 18:00:47.224119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.223953 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert\") pod \"8269db67-420d-426a-b8f6-adb240c8a6a6\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " Sep 16 18:00:47.224119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.223997 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls\") pod \"8269db67-420d-426a-b8f6-adb240c8a6a6\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " Sep 16 18:00:47.224119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.224045 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config\") pod \"8269db67-420d-426a-b8f6-adb240c8a6a6\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " Sep 16 18:00:47.224119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.224070 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location\") pod \"8269db67-420d-426a-b8f6-adb240c8a6a6\" (UID: \"8269db67-420d-426a-b8f6-adb240c8a6a6\") " Sep 16 18:00:47.224577 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.224547 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8269db67-420d-426a-b8f6-adb240c8a6a6" (UID: "8269db67-420d-426a-b8f6-adb240c8a6a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:00:47.224577 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.224544 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8269db67-420d-426a-b8f6-adb240c8a6a6" (UID: "8269db67-420d-426a-b8f6-adb240c8a6a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:00:47.224577 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.224540 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8269db67-420d-426a-b8f6-adb240c8a6a6" (UID: "8269db67-420d-426a-b8f6-adb240c8a6a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:00:47.226485 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.226457 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn" (OuterVolumeSpecName: "kube-api-access-7qvvn") pod "8269db67-420d-426a-b8f6-adb240c8a6a6" (UID: "8269db67-420d-426a-b8f6-adb240c8a6a6"). InnerVolumeSpecName "kube-api-access-7qvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:00:47.226605 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.226579 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8269db67-420d-426a-b8f6-adb240c8a6a6" (UID: "8269db67-420d-426a-b8f6-adb240c8a6a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:00:47.324902 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.324859 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvvn\" (UniqueName: \"kubernetes.io/projected/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-api-access-7qvvn\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:00:47.324902 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.324898 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:00:47.325141 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.324909 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8269db67-420d-426a-b8f6-adb240c8a6a6-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:00:47.325141 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.324944 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8269db67-420d-426a-b8f6-adb240c8a6a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:00:47.325141 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.324958 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8269db67-420d-426a-b8f6-adb240c8a6a6-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:00:47.973984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.973939 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" event={"ID":"8269db67-420d-426a-b8f6-adb240c8a6a6","Type":"ContainerDied","Data":"a5e2cc1ee6d812bd70459455e8f8d8329dc963f535f809a71104231c9a28cf47"} Sep 16 18:00:47.974474 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.973992 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" Sep 16 18:00:47.974474 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:47.974017 2536 scope.go:117] "RemoveContainer" containerID="1a590c5e967d8bb5fcba50fa396b701c4591d9e975060e41aa787d676a44ce4d" Sep 16 18:00:48.014078 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:48.013331 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 18:00:48.016234 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:48.016168 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx"] Sep 16 18:00:48.086669 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:48.086618 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-45dtx" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": context deadline exceeded" Sep 16 18:00:48.248182 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:48.247744 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:00:48.479116 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:48.479082 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" path="/var/lib/kubelet/pods/8269db67-420d-426a-b8f6-adb240c8a6a6/volumes" Sep 16 18:00:49.094673 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:49.094611 2536 scope.go:117] "RemoveContainer" containerID="62ab54f2eee57c1ccf22438f90dda7d9cd75b0dbe4654b063c513dc0955cef2b" Sep 16 18:00:49.095961 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:00:49.095930 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2530ccdb_0403_4e18_9000_e11f38956397.slice/crio-f09b8147256d236daabe386c7906aaffe6c23ea492066fbfbeb6521e950f1566 WatchSource:0}: Error finding container f09b8147256d236daabe386c7906aaffe6c23ea492066fbfbeb6521e950f1566: Status 404 returned error can't find the container with id f09b8147256d236daabe386c7906aaffe6c23ea492066fbfbeb6521e950f1566 Sep 16 18:00:49.185816 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:49.185795 2536 scope.go:117] "RemoveContainer" containerID="817e31b11c73c1596676b4a5890c91242d615c3e37ea0b4789ead28c7082e00e" Sep 16 18:00:49.987725 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:00:49.987682 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerStarted","Data":"f09b8147256d236daabe386c7906aaffe6c23ea492066fbfbeb6521e950f1566"} Sep 16 18:01:00.039576 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:00.039520 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerStarted","Data":"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474"} Sep 16 18:01:27.179463 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:27.179428 2536 generic.go:356] "Generic (PLEG): container finished" podID="2530ccdb-0403-4e18-9000-e11f38956397" containerID="5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474" exitCode=0 Sep 16 18:01:27.179935 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:27.179512 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerDied","Data":"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474"} Sep 16 18:01:28.186268 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.186236 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerStarted","Data":"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5"} Sep 16 18:01:28.186604 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.186319 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerStarted","Data":"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617"} Sep 16 18:01:28.187385 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.187359 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:01:28.187526 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.187498 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:01:28.188936 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.188891 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:01:28.209061 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:28.208999 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podStartSLOduration=38.670715551 podStartE2EDuration="43.208977833s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.538262282s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:01:28.208089522 +0000 UTC m=+2196.151411210" watchObservedRunningTime="2026-09-16 18:01:28.208977833 +0000 UTC m=+2196.152299437" Sep 16 18:01:29.193379 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.193339 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerStarted","Data":"85a2fb493fa69ff04c6288be87da4939d25d3d3ef1a6f1b16018084ab08153f8"} Sep 16 18:01:29.193847 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.193412 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerStarted","Data":"dce39c3012325a7a0faf6a1830d77e6563c686cf6d500b15d11af7c44ca85dcb"} Sep 16 18:01:29.193847 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.193471 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:29.193847 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.193626 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:29.193847 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.193677 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:01:29.195337 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.195311 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 18:01:29.215705 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:29.215652 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podStartSLOduration=1.855983378 podStartE2EDuration="1m59.215636854s" totalImagesPullingTime="1m53.652062495s" totalInitContainerRuntime="3.707590981s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:01:29.213723598 +0000 UTC m=+2197.157045221" watchObservedRunningTime="2026-09-16 18:01:29.215636854 +0000 UTC m=+2197.158958454" Sep 16 18:01:30.199118 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:30.199079 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 18:01:34.198271 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:34.198238 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:01:34.198956 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:34.198927 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:01:35.204180 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:35.204149 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:35.204985 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:35.204968 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:42.814804 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:42.814764 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 18:01:42.815318 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:42.815105 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" containerID="cri-o://dce39c3012325a7a0faf6a1830d77e6563c686cf6d500b15d11af7c44ca85dcb" gracePeriod=30 Sep 16 18:01:42.815318 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:42.815212 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kube-rbac-proxy" containerID="cri-o://85a2fb493fa69ff04c6288be87da4939d25d3d3ef1a6f1b16018084ab08153f8" gracePeriod=30 Sep 16 18:01:43.260893 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:43.260853 2536 generic.go:356] "Generic (PLEG): container finished" podID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerID="85a2fb493fa69ff04c6288be87da4939d25d3d3ef1a6f1b16018084ab08153f8" exitCode=2 Sep 16 18:01:43.261074 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:43.260944 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerDied","Data":"85a2fb493fa69ff04c6288be87da4939d25d3d3ef1a6f1b16018084ab08153f8"} Sep 16 18:01:44.199152 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:44.199110 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:01:45.200005 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.199971 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused" Sep 16 18:01:45.205280 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.205249 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 18:01:45.273504 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.273475 2536 generic.go:356] "Generic (PLEG): container finished" podID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerID="dce39c3012325a7a0faf6a1830d77e6563c686cf6d500b15d11af7c44ca85dcb" exitCode=0 Sep 16 18:01:45.273669 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.273551 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerDied","Data":"dce39c3012325a7a0faf6a1830d77e6563c686cf6d500b15d11af7c44ca85dcb"} Sep 16 18:01:45.363663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.363635 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:45.468757 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.468725 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktngk\" (UniqueName: \"kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk\") pod \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " Sep 16 18:01:45.468948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.468786 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location\") pod \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " Sep 16 18:01:45.468948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.468823 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert\") pod \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " Sep 16 18:01:45.468948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.468882 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config\") pod \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " Sep 16 18:01:45.468948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.468944 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls\") pod \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\" (UID: \"a2d3c2f6-2c36-4147-8bf9-97786486a3b3\") " Sep 16 18:01:45.469299 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.469273 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2d3c2f6-2c36-4147-8bf9-97786486a3b3" (UID: "a2d3c2f6-2c36-4147-8bf9-97786486a3b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:01:45.469386 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.469326 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2d3c2f6-2c36-4147-8bf9-97786486a3b3" (UID: "a2d3c2f6-2c36-4147-8bf9-97786486a3b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:01:45.469386 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.469334 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2d3c2f6-2c36-4147-8bf9-97786486a3b3" (UID: "a2d3c2f6-2c36-4147-8bf9-97786486a3b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:01:45.471032 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.470998 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk" (OuterVolumeSpecName: "kube-api-access-ktngk") pod "a2d3c2f6-2c36-4147-8bf9-97786486a3b3" (UID: "a2d3c2f6-2c36-4147-8bf9-97786486a3b3"). InnerVolumeSpecName "kube-api-access-ktngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:01:45.471136 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.471050 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2d3c2f6-2c36-4147-8bf9-97786486a3b3" (UID: "a2d3c2f6-2c36-4147-8bf9-97786486a3b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:01:45.570132 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.570096 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:01:45.570132 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.570127 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:01:45.570132 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.570139 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktngk\" (UniqueName: \"kubernetes.io/projected/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kube-api-access-ktngk\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:01:45.570132 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.570148 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:01:45.570408 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:45.570157 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d3c2f6-2c36-4147-8bf9-97786486a3b3-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:01:46.280058 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.280032 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" Sep 16 18:01:46.280447 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.280031 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx" event={"ID":"a2d3c2f6-2c36-4147-8bf9-97786486a3b3","Type":"ContainerDied","Data":"feaa1e14566f618e1841e6b94303d3cee90e4c75daf6af9efcc662c05c615bce"} Sep 16 18:01:46.280447 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.280172 2536 scope.go:117] "RemoveContainer" containerID="85a2fb493fa69ff04c6288be87da4939d25d3d3ef1a6f1b16018084ab08153f8" Sep 16 18:01:46.289469 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.289447 2536 scope.go:117] "RemoveContainer" containerID="dce39c3012325a7a0faf6a1830d77e6563c686cf6d500b15d11af7c44ca85dcb" Sep 16 18:01:46.297361 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.297337 2536 scope.go:117] "RemoveContainer" containerID="ce1f165f7096990047bcb0962e88763d34a8f2841091fe56171846ac33844806" Sep 16 18:01:46.304507 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.304480 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 18:01:46.309186 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.309160 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-pq2fx"] Sep 16 18:01:46.477569 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:46.477527 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" path="/var/lib/kubelet/pods/a2d3c2f6-2c36-4147-8bf9-97786486a3b3/volumes" Sep 16 18:01:48.011270 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.011230 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:01:48.012110 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012057 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" Sep 16 18:01:48.012110 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012084 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" Sep 16 18:01:48.012110 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012098 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" Sep 16 18:01:48.012110 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012107 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012145 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="storage-initializer" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012155 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="storage-initializer" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012171 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="storage-initializer" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012179 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="storage-initializer" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012196 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012204 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012214 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kube-rbac-proxy" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012222 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kube-rbac-proxy" Sep 16 18:01:48.012416 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012416 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kserve-container" Sep 16 18:01:48.012938 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012438 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kube-rbac-proxy" Sep 16 18:01:48.012938 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012453 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="8269db67-420d-426a-b8f6-adb240c8a6a6" containerName="kserve-container" Sep 16 18:01:48.012938 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.012473 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d3c2f6-2c36-4147-8bf9-97786486a3b3" containerName="kube-rbac-proxy" Sep 16 18:01:48.017713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.017692 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.020159 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.020138 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\"" Sep 16 18:01:48.020266 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.020176 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\"" Sep 16 18:01:48.024157 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.024104 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:01:48.093783 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.093721 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.093983 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.093840 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.093983 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.093905 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.094061 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.093984 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b859\" (UniqueName: \"kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.094061 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.094016 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.194850 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.194812 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b859\" (UniqueName: \"kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195083 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.194868 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195083 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.194899 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195083 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.194958 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195083 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.195029 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195083 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:01:48.195034 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found Sep 16 18:01:48.195355 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:01:48.195102 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls podName:ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45 nodeName:}" failed. No retries permitted until 2026-09-16 18:01:48.695082999 +0000 UTC m=+2216.638404595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-8jqg4" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45") : secret "isvc-xgboost-predictor-serving-cert" not found Sep 16 18:01:48.195548 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.195529 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195756 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.195736 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.195907 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.195884 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.205948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.205890 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b859\" (UniqueName: \"kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.701077 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.701042 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.703550 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.703517 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8jqg4\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:48.930321 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:48.930270 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:01:49.206067 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:49.205945 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:01:49.208730 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:01:49.208704 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3dcba2_5af4_4c58_a2e4_b30e5fc79e45.slice/crio-c2687ba4ffa76fd53ea7676021cc3dcf99bcefb4290e0fe9c7c9329f62c2f230 WatchSource:0}: Error finding container c2687ba4ffa76fd53ea7676021cc3dcf99bcefb4290e0fe9c7c9329f62c2f230: Status 404 returned error can't find the container with id c2687ba4ffa76fd53ea7676021cc3dcf99bcefb4290e0fe9c7c9329f62c2f230 Sep 16 18:01:49.297607 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:49.297570 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerStarted","Data":"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3"} Sep 16 18:01:49.297776 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:49.297646 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerStarted","Data":"c2687ba4ffa76fd53ea7676021cc3dcf99bcefb4290e0fe9c7c9329f62c2f230"} Sep 16 18:01:53.318365 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:53.318331 2536 generic.go:356] "Generic (PLEG): container finished" podID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerID="e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3" exitCode=0 Sep 16 18:01:53.318886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:53.318373 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerDied","Data":"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3"} Sep 16 18:01:53.319798 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:53.319778 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:01:54.199121 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:01:54.199078 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:04.198894 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:04.198844 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:12.420570 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:12.420530 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerStarted","Data":"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3"} Sep 16 18:02:12.421029 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:12.420595 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerStarted","Data":"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4"} Sep 16 18:02:12.421029 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:12.420786 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:02:12.444628 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:12.444582 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podStartSLOduration=2.573479168 podStartE2EDuration="25.444567188s" totalImagesPullingTime="18.911716671s" totalInitContainerRuntime="3.959371349s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:02:12.442049616 +0000 UTC m=+2240.385371230" watchObservedRunningTime="2026-09-16 18:02:12.444567188 +0000 UTC m=+2240.387888790" Sep 16 18:02:13.425005 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:13.424966 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:02:13.426478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:13.426447 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:14.198988 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:14.198944 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:14.429614 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:14.429576 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:19.434729 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:19.434635 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:02:19.435355 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:19.435326 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:24.199732 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:24.199692 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:29.435642 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:29.435605 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:34.198970 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:34.198907 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:39.436076 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:39.436038 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:44.199045 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:44.199005 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 18:02:49.436129 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:49.436092 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:02:54.199740 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:54.199707 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:02:59.435782 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:02:59.435745 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:03:09.435822 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:09.435775 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:03:19.436074 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:19.436042 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:03:28.358195 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:28.358160 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:03:28.358650 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:28.358390 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" containerID="cri-o://0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4" gracePeriod=30 Sep 16 18:03:28.358650 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:28.358438 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kube-rbac-proxy" containerID="cri-o://97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3" gracePeriod=30 Sep 16 18:03:28.771758 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:28.771725 2536 generic.go:356] "Generic (PLEG): container finished" podID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerID="97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3" exitCode=2 Sep 16 18:03:28.771952 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:28.771797 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerDied","Data":"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3"} Sep 16 18:03:29.430487 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:29.430441 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused" Sep 16 18:03:29.435830 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:29.435792 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused" Sep 16 18:03:32.209967 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.209934 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:03:32.314765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314563 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location\") pod \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " Sep 16 18:03:32.314765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314629 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config\") pod \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " Sep 16 18:03:32.314765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314657 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert\") pod \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " Sep 16 18:03:32.314765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314681 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") pod \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " Sep 16 18:03:32.314765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314767 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b859\" (UniqueName: \"kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859\") pod \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\" (UID: \"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45\") " Sep 16 18:03:32.315191 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.314974 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:03:32.315191 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.315038 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:03:32.315191 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.315053 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:03:32.316965 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.316932 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:03:32.317055 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.317005 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859" (OuterVolumeSpecName: "kube-api-access-8b859") pod "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" (UID: "ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45"). InnerVolumeSpecName "kube-api-access-8b859". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:03:32.415877 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.415832 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b859\" (UniqueName: \"kubernetes.io/projected/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-api-access-8b859\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:03:32.415877 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.415870 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:03:32.415877 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.415885 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:03:32.416158 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.415897 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:03:32.416158 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.415910 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:03:32.791778 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.791742 2536 generic.go:356] "Generic (PLEG): container finished" podID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerID="0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4" exitCode=0 Sep 16 18:03:32.791966 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.791808 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" Sep 16 18:03:32.791966 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.791826 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerDied","Data":"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4"} Sep 16 18:03:32.791966 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.791892 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4" event={"ID":"ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45","Type":"ContainerDied","Data":"c2687ba4ffa76fd53ea7676021cc3dcf99bcefb4290e0fe9c7c9329f62c2f230"} Sep 16 18:03:32.791966 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.791941 2536 scope.go:117] "RemoveContainer" containerID="97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3" Sep 16 18:03:32.800333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.800301 2536 scope.go:117] "RemoveContainer" containerID="0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4" Sep 16 18:03:32.807861 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.807844 2536 scope.go:117] "RemoveContainer" containerID="e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3" Sep 16 18:03:32.815708 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.815684 2536 scope.go:117] "RemoveContainer" containerID="97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3" Sep 16 18:03:32.816051 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:03:32.816034 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3\": container with ID starting with 97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3 not found: ID does not exist" containerID="97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3" Sep 16 18:03:32.816126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816058 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3"} err="failed to get container status \"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3\": rpc error: code = NotFound desc = could not find container \"97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3\": container with ID starting with 97b30eb4f0a9e0e83ebb30f4c8c1b2f0482051705851ef85e48bb20b5bb234c3 not found: ID does not exist" Sep 16 18:03:32.816126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816075 2536 scope.go:117] "RemoveContainer" containerID="0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4" Sep 16 18:03:32.816365 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:03:32.816332 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4\": container with ID starting with 0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4 not found: ID does not exist" containerID="0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4" Sep 16 18:03:32.816436 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816362 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4"} err="failed to get container status \"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4\": rpc error: code = NotFound desc = could not find container \"0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4\": container with ID starting with 0ec09fc42ed698e5c20a272afb970b1672a647babebbeff8551861491fe9bab4 not found: ID does not exist" Sep 16 18:03:32.816436 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816387 2536 scope.go:117] "RemoveContainer" containerID="e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3" Sep 16 18:03:32.816638 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:03:32.816613 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3\": container with ID starting with e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3 not found: ID does not exist" containerID="e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3" Sep 16 18:03:32.816697 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816636 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3"} err="failed to get container status \"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3\": rpc error: code = NotFound desc = could not find container \"e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3\": container with ID starting with e07deee5d2c8b5d599a3948429ceff1de92b70b2c73fce0a277cc2572a745ae3 not found: ID does not exist" Sep 16 18:03:32.816996 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.816968 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:03:32.819998 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:32.819972 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8jqg4"] Sep 16 18:03:34.478165 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:03:34.478132 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" path="/var/lib/kubelet/pods/ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45/volumes" Sep 16 18:04:15.804463 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:15.804426 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:04:15.805226 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:15.804683 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" containerID="cri-o://32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617" gracePeriod=30 Sep 16 18:04:15.805226 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:15.804730 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kube-rbac-proxy" containerID="cri-o://a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5" gracePeriod=30 Sep 16 18:04:15.995397 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:15.995364 2536 generic.go:356] "Generic (PLEG): container finished" podID="2530ccdb-0403-4e18-9000-e11f38956397" containerID="a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5" exitCode=2 Sep 16 18:04:15.995550 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:15.995453 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerDied","Data":"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5"} Sep 16 18:04:19.194726 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:19.194675 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused" Sep 16 18:04:20.882950 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.882927 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:04:20.921548 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.921515 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkgtk\" (UniqueName: \"kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk\") pod \"2530ccdb-0403-4e18-9000-e11f38956397\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " Sep 16 18:04:20.921749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.921563 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert\") pod \"2530ccdb-0403-4e18-9000-e11f38956397\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " Sep 16 18:04:20.921749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.921600 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location\") pod \"2530ccdb-0403-4e18-9000-e11f38956397\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " Sep 16 18:04:20.921749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.921638 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls\") pod \"2530ccdb-0403-4e18-9000-e11f38956397\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " Sep 16 18:04:20.921749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.921677 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config\") pod \"2530ccdb-0403-4e18-9000-e11f38956397\" (UID: \"2530ccdb-0403-4e18-9000-e11f38956397\") " Sep 16 18:04:20.922048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.922016 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2530ccdb-0403-4e18-9000-e11f38956397" (UID: "2530ccdb-0403-4e18-9000-e11f38956397"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:04:20.922495 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.922461 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2530ccdb-0403-4e18-9000-e11f38956397" (UID: "2530ccdb-0403-4e18-9000-e11f38956397"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:04:20.922602 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.922519 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2530ccdb-0403-4e18-9000-e11f38956397" (UID: "2530ccdb-0403-4e18-9000-e11f38956397"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:04:20.924100 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.924067 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk" (OuterVolumeSpecName: "kube-api-access-gkgtk") pod "2530ccdb-0403-4e18-9000-e11f38956397" (UID: "2530ccdb-0403-4e18-9000-e11f38956397"). InnerVolumeSpecName "kube-api-access-gkgtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:04:20.924517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:20.924484 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2530ccdb-0403-4e18-9000-e11f38956397" (UID: "2530ccdb-0403-4e18-9000-e11f38956397"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:04:21.024574 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.024540 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgtk\" (UniqueName: \"kubernetes.io/projected/2530ccdb-0403-4e18-9000-e11f38956397-kube-api-access-gkgtk\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:04:21.024574 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.024572 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:04:21.024766 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.024586 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2530ccdb-0403-4e18-9000-e11f38956397-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:04:21.024766 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.024601 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2530ccdb-0403-4e18-9000-e11f38956397-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:04:21.024766 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.024613 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2530ccdb-0403-4e18-9000-e11f38956397-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:04:21.028159 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.028130 2536 generic.go:356] "Generic (PLEG): container finished" podID="2530ccdb-0403-4e18-9000-e11f38956397" containerID="32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617" exitCode=0 Sep 16 18:04:21.028287 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.028190 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" Sep 16 18:04:21.028287 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.028197 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerDied","Data":"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617"} Sep 16 18:04:21.028287 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.028272 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp" event={"ID":"2530ccdb-0403-4e18-9000-e11f38956397","Type":"ContainerDied","Data":"f09b8147256d236daabe386c7906aaffe6c23ea492066fbfbeb6521e950f1566"} Sep 16 18:04:21.028388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.028299 2536 scope.go:117] "RemoveContainer" containerID="a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5" Sep 16 18:04:21.037326 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.037308 2536 scope.go:117] "RemoveContainer" containerID="32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617" Sep 16 18:04:21.044543 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.044525 2536 scope.go:117] "RemoveContainer" containerID="5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474" Sep 16 18:04:21.053907 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.053876 2536 scope.go:117] "RemoveContainer" containerID="a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5" Sep 16 18:04:21.054258 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:04:21.054239 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5\": container with ID starting with a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5 not found: ID does not exist" containerID="a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5" Sep 16 18:04:21.054366 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.054272 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5"} err="failed to get container status \"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5\": rpc error: code = NotFound desc = could not find container \"a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5\": container with ID starting with a61ab51c1f6f21299ccd33d818e099aed75f150d24f8996dbd80383a33db82d5 not found: ID does not exist" Sep 16 18:04:21.054366 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.054299 2536 scope.go:117] "RemoveContainer" containerID="32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617" Sep 16 18:04:21.054565 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:04:21.054545 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617\": container with ID starting with 32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617 not found: ID does not exist" containerID="32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617" Sep 16 18:04:21.054614 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.054576 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617"} err="failed to get container status \"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617\": rpc error: code = NotFound desc = could not find container \"32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617\": container with ID starting with 32959ebbc3e6b2d120287a8b3b39dcacbfb1482a7e01c54d9748f9e554fe2617 not found: ID does not exist" Sep 16 18:04:21.054614 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.054592 2536 scope.go:117] "RemoveContainer" containerID="5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474" Sep 16 18:04:21.054860 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:04:21.054844 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474\": container with ID starting with 5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474 not found: ID does not exist" containerID="5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474" Sep 16 18:04:21.054903 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.054864 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474"} err="failed to get container status \"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474\": rpc error: code = NotFound desc = could not find container \"5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474\": container with ID starting with 5faf7dedbc4d400b429500bac11ec99365603a6327aecb51f007d57a3f521474 not found: ID does not exist" Sep 16 18:04:21.059516 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.059492 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:04:21.071709 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.071675 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-frhkp"] Sep 16 18:04:21.117277 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117244 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:04:21.117564 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117552 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kube-rbac-proxy" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117566 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kube-rbac-proxy" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117582 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117588 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117595 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="storage-initializer" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117602 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="storage-initializer" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117612 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kube-rbac-proxy" Sep 16 18:04:21.117617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117617 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kube-rbac-proxy" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117623 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117628 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117635 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="storage-initializer" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117640 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="storage-initializer" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117702 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kserve-container" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117709 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3dcba2-5af4-4c58-a2e4-b30e5fc79e45" containerName="kube-rbac-proxy" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117717 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kube-rbac-proxy" Sep 16 18:04:21.117828 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.117724 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2530ccdb-0403-4e18-9000-e11f38956397" containerName="kserve-container" Sep 16 18:04:21.120831 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.120807 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.125066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.125039 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.125247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.125095 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.125247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.125126 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.125247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.125149 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.125247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.125197 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.126152 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126096 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:04:21.126152 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126096 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:04:21.126339 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126267 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\"" Sep 16 18:04:21.126396 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126351 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 16 18:04:21.126443 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126421 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:04:21.126550 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126537 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:04:21.126627 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.126607 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:04:21.144027 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.143993 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:04:21.218091 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.218062 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:04:21.220463 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.220443 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.223712 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.223695 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 16 18:04:21.225945 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.225902 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.226029 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.225975 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.226029 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226005 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzft\" (UniqueName: \"kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.226104 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226038 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.226104 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226083 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226175 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226160 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226235 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226214 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226349 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226256 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226349 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226288 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.226349 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226320 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226534 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:04:21.226438 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found Sep 16 18:04:21.226534 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:04:21.226494 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls podName:9574c29c-a085-43a5-b955-0a2241ab7e87 nodeName:}" failed. No retries permitted until 2026-09-16 18:04:21.726474616 +0000 UTC m=+2369.669796203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls") pod "isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found Sep 16 18:04:21.226692 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226668 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.226940 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.226895 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.227075 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.227056 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.239482 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.239457 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:04:21.240330 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.240309 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.327018 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.326908 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327018 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327018 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327254 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327048 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327254 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327071 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327254 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327085 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzft\" (UniqueName: \"kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327423 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327385 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327763 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327711 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.327763 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.327730 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.329932 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.329902 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.340594 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.340559 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzft\" (UniqueName: \"kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft\") pod \"isvc-canary-stop-predictor-59fbfd5bc-ht6m2\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.530790 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.530741 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:21.731268 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.731228 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.733771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.733738 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:21.842798 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:21.842769 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:04:21.843671 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:04:21.843646 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58368f5a_9f00_4272_9dae_31ec4d70218f.slice/crio-f02783441c63f781d92c84ee4062bc65e69085c3a4248759fdbb2f5ffaf9a5f4 WatchSource:0}: Error finding container f02783441c63f781d92c84ee4062bc65e69085c3a4248759fdbb2f5ffaf9a5f4: Status 404 returned error can't find the container with id f02783441c63f781d92c84ee4062bc65e69085c3a4248759fdbb2f5ffaf9a5f4 Sep 16 18:04:22.031881 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:22.031842 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:22.036070 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:22.036022 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerStarted","Data":"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574"} Sep 16 18:04:22.036214 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:22.036099 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerStarted","Data":"f02783441c63f781d92c84ee4062bc65e69085c3a4248759fdbb2f5ffaf9a5f4"} Sep 16 18:04:22.323431 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:22.323353 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:04:22.325653 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:04:22.325623 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9574c29c_a085_43a5_b955_0a2241ab7e87.slice/crio-c06b979772e27b7a7fc071349ce37f85815153f2752b47146843b1d6ef81d16a WatchSource:0}: Error finding container c06b979772e27b7a7fc071349ce37f85815153f2752b47146843b1d6ef81d16a: Status 404 returned error can't find the container with id c06b979772e27b7a7fc071349ce37f85815153f2752b47146843b1d6ef81d16a Sep 16 18:04:22.478881 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:22.478843 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2530ccdb-0403-4e18-9000-e11f38956397" path="/var/lib/kubelet/pods/2530ccdb-0403-4e18-9000-e11f38956397/volumes" Sep 16 18:04:23.041487 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:23.041452 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerStarted","Data":"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39"} Sep 16 18:04:23.041965 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:23.041526 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerStarted","Data":"c06b979772e27b7a7fc071349ce37f85815153f2752b47146843b1d6ef81d16a"} Sep 16 18:04:26.056829 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:26.056796 2536 generic.go:356] "Generic (PLEG): container finished" podID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerID="310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574" exitCode=0 Sep 16 18:04:26.057264 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:26.056849 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerDied","Data":"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574"} Sep 16 18:04:27.063336 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.063301 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerStarted","Data":"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9"} Sep 16 18:04:27.063749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.063364 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerStarted","Data":"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d"} Sep 16 18:04:27.063749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.063564 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:27.064850 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.064829 2536 generic.go:356] "Generic (PLEG): container finished" podID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerID="2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39" exitCode=0 Sep 16 18:04:27.064907 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.064879 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerDied","Data":"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39"} Sep 16 18:04:27.088736 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:27.088685 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podStartSLOduration=1.9614732830000001 podStartE2EDuration="6.088668713s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.12719543s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:27.087885559 +0000 UTC m=+2375.031207181" watchObservedRunningTime="2026-09-16 18:04:27.088668713 +0000 UTC m=+2375.031990316" Sep 16 18:04:28.071704 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.071670 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerStarted","Data":"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01"} Sep 16 18:04:28.072215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.071731 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerStarted","Data":"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4"} Sep 16 18:04:28.072215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.071890 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:28.072215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.072091 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:28.073475 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.073451 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:04:28.349970 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:28.349839 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podStartSLOduration=3.392602252 podStartE2EDuration="7.34982261s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957220358s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:28.348838041 +0000 UTC m=+2376.292159644" watchObservedRunningTime="2026-09-16 18:04:28.34982261 +0000 UTC m=+2376.293144213" Sep 16 18:04:29.077428 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:29.077378 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:04:29.077856 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:29.077569 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:29.078986 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:29.078964 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:04:30.081716 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:30.081677 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:04:34.082047 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:34.082019 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:04:34.082547 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:34.082517 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:04:35.086457 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:35.086429 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:04:35.086982 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:35.086957 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:04:44.083187 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:44.083136 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:04:45.087431 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:45.087393 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:04:52.603841 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:52.603812 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:04:52.610312 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:52.610287 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:04:54.083371 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:54.083331 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:04:55.087081 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:04:55.087046 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:04.082626 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:04.082589 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:05:05.087435 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:05.087397 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:14.082993 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:14.082954 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:05:15.087708 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:15.087619 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:24.083150 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:24.083111 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:05:25.087662 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:25.087625 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:34.082723 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:34.082680 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:05:35.087556 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:35.087514 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:35.472832 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:35.472797 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused" Sep 16 18:05:45.087742 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.087712 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:05:45.253414 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.253378 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:05:45.253632 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.253610 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" containerID="cri-o://2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d" gracePeriod=30 Sep 16 18:05:45.253787 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.253749 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kube-rbac-proxy" containerID="cri-o://cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9" gracePeriod=30 Sep 16 18:05:45.254708 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.254688 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:05:45.427274 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.427240 2536 generic.go:356] "Generic (PLEG): container finished" podID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerID="cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9" exitCode=2 Sep 16 18:05:45.427441 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:45.427286 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerDied","Data":"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9"} Sep 16 18:05:49.077662 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:49.077618 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused" Sep 16 18:05:50.206602 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.206575 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:05:50.285608 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.285516 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config\") pod \"58368f5a-9f00-4272-9dae-31ec4d70218f\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " Sep 16 18:05:50.285608 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.285577 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert\") pod \"58368f5a-9f00-4272-9dae-31ec4d70218f\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " Sep 16 18:05:50.285608 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.285605 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls\") pod \"58368f5a-9f00-4272-9dae-31ec4d70218f\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " Sep 16 18:05:50.285905 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.285663 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location\") pod \"58368f5a-9f00-4272-9dae-31ec4d70218f\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " Sep 16 18:05:50.285905 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.285695 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlzft\" (UniqueName: \"kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft\") pod \"58368f5a-9f00-4272-9dae-31ec4d70218f\" (UID: \"58368f5a-9f00-4272-9dae-31ec4d70218f\") " Sep 16 18:05:50.286053 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.286012 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58368f5a-9f00-4272-9dae-31ec4d70218f" (UID: "58368f5a-9f00-4272-9dae-31ec4d70218f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:05:50.286053 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.286015 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58368f5a-9f00-4272-9dae-31ec4d70218f" (UID: "58368f5a-9f00-4272-9dae-31ec4d70218f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:05:50.286154 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.286049 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58368f5a-9f00-4272-9dae-31ec4d70218f" (UID: "58368f5a-9f00-4272-9dae-31ec4d70218f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:05:50.287853 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.287828 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58368f5a-9f00-4272-9dae-31ec4d70218f" (UID: "58368f5a-9f00-4272-9dae-31ec4d70218f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:05:50.287970 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.287851 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft" (OuterVolumeSpecName: "kube-api-access-rlzft") pod "58368f5a-9f00-4272-9dae-31ec4d70218f" (UID: "58368f5a-9f00-4272-9dae-31ec4d70218f"). InnerVolumeSpecName "kube-api-access-rlzft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:05:50.386685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.386644 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58368f5a-9f00-4272-9dae-31ec4d70218f-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:50.386685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.386676 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzft\" (UniqueName: \"kubernetes.io/projected/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-api-access-rlzft\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:50.386685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.386685 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:50.386685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.386695 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58368f5a-9f00-4272-9dae-31ec4d70218f-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:50.386999 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.386705 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58368f5a-9f00-4272-9dae-31ec4d70218f-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:50.451458 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.451427 2536 generic.go:356] "Generic (PLEG): container finished" podID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerID="2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d" exitCode=0 Sep 16 18:05:50.451618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.451492 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" Sep 16 18:05:50.451618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.451509 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerDied","Data":"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d"} Sep 16 18:05:50.451618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.451575 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2" event={"ID":"58368f5a-9f00-4272-9dae-31ec4d70218f","Type":"ContainerDied","Data":"f02783441c63f781d92c84ee4062bc65e69085c3a4248759fdbb2f5ffaf9a5f4"} Sep 16 18:05:50.451618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.451603 2536 scope.go:117] "RemoveContainer" containerID="cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9" Sep 16 18:05:50.460585 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.460562 2536 scope.go:117] "RemoveContainer" containerID="2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d" Sep 16 18:05:50.468319 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.468301 2536 scope.go:117] "RemoveContainer" containerID="310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574" Sep 16 18:05:50.476024 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.476000 2536 scope.go:117] "RemoveContainer" containerID="cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9" Sep 16 18:05:50.476545 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:50.476519 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9\": container with ID starting with cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9 not found: ID does not exist" containerID="cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9" Sep 16 18:05:50.476685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.476639 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9"} err="failed to get container status \"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9\": rpc error: code = NotFound desc = could not find container \"cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9\": container with ID starting with cafbbf74f1ecf54c891f4f4b2329f3590399d61f4eaa0edb8e1ec5c4963369e9 not found: ID does not exist" Sep 16 18:05:50.476685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.476678 2536 scope.go:117] "RemoveContainer" containerID="2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d" Sep 16 18:05:50.477103 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:50.477063 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d\": container with ID starting with 2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d not found: ID does not exist" containerID="2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d" Sep 16 18:05:50.477208 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.477115 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d"} err="failed to get container status \"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d\": rpc error: code = NotFound desc = could not find container \"2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d\": container with ID starting with 2c917930263127f07fd99159b5b445bd7ef33837c377ad0cf76c3eae5130fd5d not found: ID does not exist" Sep 16 18:05:50.477208 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.477145 2536 scope.go:117] "RemoveContainer" containerID="310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574" Sep 16 18:05:50.477477 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:50.477458 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574\": container with ID starting with 310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574 not found: ID does not exist" containerID="310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574" Sep 16 18:05:50.477579 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.477483 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574"} err="failed to get container status \"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574\": rpc error: code = NotFound desc = could not find container \"310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574\": container with ID starting with 310bded1f22d3740f6a07cdc162e20abbc8eddae9635358d620a82436be4f574 not found: ID does not exist" Sep 16 18:05:50.478863 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.478843 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:05:50.489782 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:50.489758 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-ht6m2"] Sep 16 18:05:51.116034 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:51.116000 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:05:51.116312 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:51.116278 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" containerID="cri-o://ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4" gracePeriod=30 Sep 16 18:05:51.116312 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:51.116313 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kube-rbac-proxy" containerID="cri-o://edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01" gracePeriod=30 Sep 16 18:05:51.458805 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:51.458714 2536 generic.go:356] "Generic (PLEG): container finished" podID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerID="edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01" exitCode=2 Sep 16 18:05:51.458805 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:51.458788 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerDied","Data":"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01"} Sep 16 18:05:52.478095 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:52.478067 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" path="/var/lib/kubelet/pods/58368f5a-9f00-4272-9dae-31ec4d70218f/volumes" Sep 16 18:05:55.082804 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:55.082760 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused" Sep 16 18:05:55.087184 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:55.087153 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused" Sep 16 18:05:56.274268 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.274241 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:05:56.439749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.439659 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config\") pod \"9574c29c-a085-43a5-b955-0a2241ab7e87\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " Sep 16 18:05:56.439749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.439700 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert\") pod \"9574c29c-a085-43a5-b955-0a2241ab7e87\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " Sep 16 18:05:56.439749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.439727 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") pod \"9574c29c-a085-43a5-b955-0a2241ab7e87\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " Sep 16 18:05:56.440071 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.439759 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk\") pod \"9574c29c-a085-43a5-b955-0a2241ab7e87\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " Sep 16 18:05:56.440071 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.439879 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location\") pod \"9574c29c-a085-43a5-b955-0a2241ab7e87\" (UID: \"9574c29c-a085-43a5-b955-0a2241ab7e87\") " Sep 16 18:05:56.440194 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.440152 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9574c29c-a085-43a5-b955-0a2241ab7e87" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:05:56.440194 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.440162 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9574c29c-a085-43a5-b955-0a2241ab7e87" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:05:56.440194 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.440181 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9574c29c-a085-43a5-b955-0a2241ab7e87" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:05:56.441873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.441848 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9574c29c-a085-43a5-b955-0a2241ab7e87" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:05:56.442003 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.441902 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk" (OuterVolumeSpecName: "kube-api-access-ptqdk") pod "9574c29c-a085-43a5-b955-0a2241ab7e87" (UID: "9574c29c-a085-43a5-b955-0a2241ab7e87"). InnerVolumeSpecName "kube-api-access-ptqdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:05:56.482311 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.482273 2536 generic.go:356] "Generic (PLEG): container finished" podID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerID="ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4" exitCode=0 Sep 16 18:05:56.482488 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.482339 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerDied","Data":"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4"} Sep 16 18:05:56.482488 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.482348 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" Sep 16 18:05:56.482488 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.482387 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7" event={"ID":"9574c29c-a085-43a5-b955-0a2241ab7e87","Type":"ContainerDied","Data":"c06b979772e27b7a7fc071349ce37f85815153f2752b47146843b1d6ef81d16a"} Sep 16 18:05:56.482488 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.482412 2536 scope.go:117] "RemoveContainer" containerID="edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01" Sep 16 18:05:56.492172 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.492152 2536 scope.go:117] "RemoveContainer" containerID="ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4" Sep 16 18:05:56.499851 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.499832 2536 scope.go:117] "RemoveContainer" containerID="2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39" Sep 16 18:05:56.507638 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.507613 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:05:56.507895 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.507874 2536 scope.go:117] "RemoveContainer" containerID="edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01" Sep 16 18:05:56.508241 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:56.508213 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01\": container with ID starting with edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01 not found: ID does not exist" containerID="edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01" Sep 16 18:05:56.508303 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.508250 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01"} err="failed to get container status \"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01\": rpc error: code = NotFound desc = could not find container \"edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01\": container with ID starting with edf1b31b7521a61472e8028f46db5bbc555eec3f64f5532ca43dcb397c90df01 not found: ID does not exist" Sep 16 18:05:56.508303 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.508268 2536 scope.go:117] "RemoveContainer" containerID="ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4" Sep 16 18:05:56.508581 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:56.508555 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4\": container with ID starting with ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4 not found: ID does not exist" containerID="ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4" Sep 16 18:05:56.508690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.508600 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4"} err="failed to get container status \"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4\": rpc error: code = NotFound desc = could not find container \"ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4\": container with ID starting with ed0585fbc13ce9920846b4bc1303c3cb7dcb7026bd7c0c36afcc485a3a1814a4 not found: ID does not exist" Sep 16 18:05:56.508690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.508625 2536 scope.go:117] "RemoveContainer" containerID="2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39" Sep 16 18:05:56.509098 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:05:56.509079 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39\": container with ID starting with 2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39 not found: ID does not exist" containerID="2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39" Sep 16 18:05:56.509195 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.509101 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39"} err="failed to get container status \"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39\": rpc error: code = NotFound desc = could not find container \"2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39\": container with ID starting with 2f2e8acd4e9dcfc18cf04729cd6e6c32ed9d4653400942b8ba582a405b2cdc39 not found: ID does not exist" Sep 16 18:05:56.511271 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.511250 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-4gfh7"] Sep 16 18:05:56.541489 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.541459 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9574c29c-a085-43a5-b955-0a2241ab7e87-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:56.541489 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.541490 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:56.541652 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.541499 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9574c29c-a085-43a5-b955-0a2241ab7e87-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:56.541652 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.541509 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9574c29c-a085-43a5-b955-0a2241ab7e87-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:56.541652 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:56.541553 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/9574c29c-a085-43a5-b955-0a2241ab7e87-kube-api-access-ptqdk\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:05:58.478070 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:05:58.478038 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" path="/var/lib/kubelet/pods/9574c29c-a085-43a5-b955-0a2241ab7e87/volumes" Sep 16 18:06:01.344833 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.344780 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345212 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kube-rbac-proxy" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345226 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kube-rbac-proxy" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345234 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kube-rbac-proxy" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345239 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kube-rbac-proxy" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345250 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345256 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345276 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="storage-initializer" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345282 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="storage-initializer" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345292 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="storage-initializer" Sep 16 18:06:01.345296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345297 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="storage-initializer" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345304 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345310 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345361 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kube-rbac-proxy" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345367 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="58368f5a-9f00-4272-9dae-31ec4d70218f" containerName="kserve-container" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345374 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kserve-container" Sep 16 18:06:01.345616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.345380 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="9574c29c-a085-43a5-b955-0a2241ab7e87" containerName="kube-rbac-proxy" Sep 16 18:06:01.348395 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.348377 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.351722 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.351698 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:06:01.352024 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352004 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:06:01.352130 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352074 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:06:01.352729 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352708 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:06:01.352822 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352709 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:06:01.352822 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352712 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\"" Sep 16 18:06:01.352822 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.352710 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\"" Sep 16 18:06:01.363026 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.363002 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:06:01.482942 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.482889 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.483125 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.482948 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.483125 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.483007 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg294\" (UniqueName: \"kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.483125 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.483087 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.483252 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.483136 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584370 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584319 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584370 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584371 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584645 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584425 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584645 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584443 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584645 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584458 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg294\" (UniqueName: \"kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.584842 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.584821 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.585185 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.585167 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.585242 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.585177 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.586932 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.586899 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.593783 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.593759 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg294\" (UniqueName: \"kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294\") pod \"isvc-lightgbm-predictor-54f85ddf88-xl59c\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.659190 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.659084 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:01.941167 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:01.941066 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:06:01.946447 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:06:01.946419 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5d6af2_f487_42e0_9828_07cc738f0da8.slice/crio-cf4feca3194885c05dd4ef9c858504febea526586207387c2179e1fe8c790775 WatchSource:0}: Error finding container cf4feca3194885c05dd4ef9c858504febea526586207387c2179e1fe8c790775: Status 404 returned error can't find the container with id cf4feca3194885c05dd4ef9c858504febea526586207387c2179e1fe8c790775 Sep 16 18:06:02.510679 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:02.510640 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerStarted","Data":"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc"} Sep 16 18:06:02.511182 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:02.510718 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerStarted","Data":"cf4feca3194885c05dd4ef9c858504febea526586207387c2179e1fe8c790775"} Sep 16 18:06:06.529559 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:06.529522 2536 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerID="0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc" exitCode=0 Sep 16 18:06:06.529972 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:06.529598 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerDied","Data":"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc"} Sep 16 18:06:13.566974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:13.566940 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerStarted","Data":"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e"} Sep 16 18:06:13.567411 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:13.567004 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerStarted","Data":"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b"} Sep 16 18:06:13.567411 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:13.567187 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:13.595181 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:13.595126 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podStartSLOduration=2.020402632 podStartE2EDuration="12.595109461s" totalImagesPullingTime="6.584339417s" totalInitContainerRuntime="3.990367412s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:13.593188622 +0000 UTC m=+2481.536510245" watchObservedRunningTime="2026-09-16 18:06:13.595109461 +0000 UTC m=+2481.538431065" Sep 16 18:06:14.571435 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:14.571406 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:14.573150 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:14.573121 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:06:15.575973 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:15.575910 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:06:20.580824 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:20.580792 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:06:20.581467 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:20.581436 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:06:30.582379 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:30.582335 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:06:40.581772 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:40.581730 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:06:50.581865 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:06:50.581820 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:07:00.582329 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:00.582282 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:07:10.582185 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:10.582141 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:07:20.581497 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:20.581458 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused" Sep 16 18:07:30.582481 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:30.582450 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:07:31.454911 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:31.454876 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:07:31.455189 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:31.455163 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" containerID="cri-o://ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b" gracePeriod=30 Sep 16 18:07:31.455276 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:31.455195 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kube-rbac-proxy" containerID="cri-o://17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e" gracePeriod=30 Sep 16 18:07:31.905780 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:31.905739 2536 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerID="17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e" exitCode=2 Sep 16 18:07:31.906206 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:31.905815 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerDied","Data":"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e"} Sep 16 18:07:35.577033 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:35.576996 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused" Sep 16 18:07:36.211100 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.211074 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:07:36.308879 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.308767 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls\") pod \"dd5d6af2-f487-42e0-9828-07cc738f0da8\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " Sep 16 18:07:36.308879 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.308832 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config\") pod \"dd5d6af2-f487-42e0-9828-07cc738f0da8\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " Sep 16 18:07:36.309104 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.308895 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg294\" (UniqueName: \"kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294\") pod \"dd5d6af2-f487-42e0-9828-07cc738f0da8\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " Sep 16 18:07:36.309144 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309105 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location\") pod \"dd5d6af2-f487-42e0-9828-07cc738f0da8\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " Sep 16 18:07:36.309144 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309139 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert\") pod \"dd5d6af2-f487-42e0-9828-07cc738f0da8\" (UID: \"dd5d6af2-f487-42e0-9828-07cc738f0da8\") " Sep 16 18:07:36.309280 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309244 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd5d6af2-f487-42e0-9828-07cc738f0da8" (UID: "dd5d6af2-f487-42e0-9828-07cc738f0da8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:07:36.309476 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309396 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd5d6af2-f487-42e0-9828-07cc738f0da8" (UID: "dd5d6af2-f487-42e0-9828-07cc738f0da8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:07:36.309476 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309442 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:07:36.309567 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.309524 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd5d6af2-f487-42e0-9828-07cc738f0da8" (UID: "dd5d6af2-f487-42e0-9828-07cc738f0da8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:07:36.311530 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.311505 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd5d6af2-f487-42e0-9828-07cc738f0da8" (UID: "dd5d6af2-f487-42e0-9828-07cc738f0da8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:07:36.311623 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.311523 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294" (OuterVolumeSpecName: "kube-api-access-bg294") pod "dd5d6af2-f487-42e0-9828-07cc738f0da8" (UID: "dd5d6af2-f487-42e0-9828-07cc738f0da8"). InnerVolumeSpecName "kube-api-access-bg294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:07:36.410744 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.410707 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg294\" (UniqueName: \"kubernetes.io/projected/dd5d6af2-f487-42e0-9828-07cc738f0da8-kube-api-access-bg294\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:07:36.410744 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.410737 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6af2-f487-42e0-9828-07cc738f0da8-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:07:36.410744 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.410747 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5d6af2-f487-42e0-9828-07cc738f0da8-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:07:36.410744 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.410756 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5d6af2-f487-42e0-9828-07cc738f0da8-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:07:36.638372 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638286 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638697 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638716 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638738 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="storage-initializer" Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638744 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="storage-initializer" Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638760 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kube-rbac-proxy" Sep 16 18:07:36.638771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638765 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kube-rbac-proxy" Sep 16 18:07:36.639010 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638825 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kserve-container" Sep 16 18:07:36.639010 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.638834 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerName="kube-rbac-proxy" Sep 16 18:07:36.642101 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.642078 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.647157 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.647132 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\"" Sep 16 18:07:36.647157 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.647137 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\"" Sep 16 18:07:36.659017 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.658984 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:07:36.713679 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.713634 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.713873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.713746 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5km6\" (UniqueName: \"kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.713873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.713797 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.713873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.713834 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.713873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.713858 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.814465 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.814423 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.814663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.814494 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.814663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.814526 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.814663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.814554 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.814663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.814642 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5km6\" (UniqueName: \"kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.815086 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.815063 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.815250 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.815226 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.815346 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.815323 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.817130 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.817111 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.825566 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.825545 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5km6\" (UniqueName: \"kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-22v56\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.928469 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.928371 2536 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6af2-f487-42e0-9828-07cc738f0da8" containerID="ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b" exitCode=0 Sep 16 18:07:36.928469 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.928442 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" Sep 16 18:07:36.928469 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.928463 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerDied","Data":"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b"} Sep 16 18:07:36.928690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.928524 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c" event={"ID":"dd5d6af2-f487-42e0-9828-07cc738f0da8","Type":"ContainerDied","Data":"cf4feca3194885c05dd4ef9c858504febea526586207387c2179e1fe8c790775"} Sep 16 18:07:36.928690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.928553 2536 scope.go:117] "RemoveContainer" containerID="17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e" Sep 16 18:07:36.936633 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.936612 2536 scope.go:117] "RemoveContainer" containerID="ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b" Sep 16 18:07:36.943603 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.943585 2536 scope.go:117] "RemoveContainer" containerID="0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc" Sep 16 18:07:36.951000 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.950980 2536 scope.go:117] "RemoveContainer" containerID="17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e" Sep 16 18:07:36.951320 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:07:36.951300 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e\": container with ID starting with 17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e not found: ID does not exist" containerID="17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e" Sep 16 18:07:36.951400 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.951334 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e"} err="failed to get container status \"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e\": rpc error: code = NotFound desc = could not find container \"17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e\": container with ID starting with 17d6b3d1ed49549e7827fe4af3b9a2bd8c4e4677732e4ff57de47a7976588f0e not found: ID does not exist" Sep 16 18:07:36.951400 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.951357 2536 scope.go:117] "RemoveContainer" containerID="ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b" Sep 16 18:07:36.951638 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:07:36.951618 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b\": container with ID starting with ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b not found: ID does not exist" containerID="ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b" Sep 16 18:07:36.951675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.951649 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b"} err="failed to get container status \"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b\": rpc error: code = NotFound desc = could not find container \"ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b\": container with ID starting with ef34012d7201927b1e7b1d29afe02bdb4302d439c3fa29ad1d9668ccb516385b not found: ID does not exist" Sep 16 18:07:36.951675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.951665 2536 scope.go:117] "RemoveContainer" containerID="0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc" Sep 16 18:07:36.951924 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:07:36.951893 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc\": container with ID starting with 0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc not found: ID does not exist" containerID="0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc" Sep 16 18:07:36.951992 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.951931 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc"} err="failed to get container status \"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc\": rpc error: code = NotFound desc = could not find container \"0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc\": container with ID starting with 0ddfe0dbd564212e5a7d27bf4c61c683bed96972efae62e5502d4325391f1cbc not found: ID does not exist" Sep 16 18:07:36.957466 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.957437 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:36.959321 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.959298 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:07:36.969401 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:36.969379 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xl59c"] Sep 16 18:07:37.235983 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:37.235959 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:07:37.238306 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:07:37.238278 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d064ff_0a08_40f1_822b_8b73f7513956.slice/crio-31713e6db6fe66c650397cb375bda80bc28fc5180c0035980931d4b4a7151c31 WatchSource:0}: Error finding container 31713e6db6fe66c650397cb375bda80bc28fc5180c0035980931d4b4a7151c31: Status 404 returned error can't find the container with id 31713e6db6fe66c650397cb375bda80bc28fc5180c0035980931d4b4a7151c31 Sep 16 18:07:37.936632 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:37.936596 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerStarted","Data":"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6"} Sep 16 18:07:37.937137 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:37.936652 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerStarted","Data":"31713e6db6fe66c650397cb375bda80bc28fc5180c0035980931d4b4a7151c31"} Sep 16 18:07:38.479067 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:38.479038 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd5d6af2-f487-42e0-9828-07cc738f0da8" path="/var/lib/kubelet/pods/dd5d6af2-f487-42e0-9828-07cc738f0da8/volumes" Sep 16 18:07:41.956218 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:41.956183 2536 generic.go:356] "Generic (PLEG): container finished" podID="45d064ff-0a08-40f1-822b-8b73f7513956" containerID="7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6" exitCode=0 Sep 16 18:07:41.956617 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:41.956258 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerDied","Data":"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6"} Sep 16 18:07:42.964125 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:42.964092 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerStarted","Data":"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306"} Sep 16 18:07:42.964505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:42.964158 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerStarted","Data":"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426"} Sep 16 18:07:42.964505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:42.964353 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:42.993598 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:42.993541 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podStartSLOduration=3.1272381989999998 podStartE2EDuration="6.99352497s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866286771s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:07:42.993055004 +0000 UTC m=+2570.936376636" watchObservedRunningTime="2026-09-16 18:07:42.99352497 +0000 UTC m=+2570.936846573" Sep 16 18:07:43.969066 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:43.969034 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:43.970616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:43.970587 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:07:44.973399 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:44.973362 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:07:49.977899 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:49.977861 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:07:49.978460 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:49.978433 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:07:59.979237 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:07:59.979195 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:09.978590 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:09.978549 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:19.979470 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:19.979383 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:29.978755 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:29.978715 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:39.978604 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:39.978564 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:49.979131 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:49.979093 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:08:59.979072 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:08:59.979039 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:09:06.919422 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:06.919386 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:09:06.919947 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:06.919732 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" containerID="cri-o://7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426" gracePeriod=30 Sep 16 18:09:06.919947 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:06.919799 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kube-rbac-proxy" containerID="cri-o://b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306" gracePeriod=30 Sep 16 18:09:07.353342 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:07.353306 2536 generic.go:356] "Generic (PLEG): container finished" podID="45d064ff-0a08-40f1-822b-8b73f7513956" containerID="b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306" exitCode=2 Sep 16 18:09:07.353554 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:07.353369 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerDied","Data":"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306"} Sep 16 18:09:09.974009 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:09.973967 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused" Sep 16 18:09:09.979401 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:09.979371 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused" Sep 16 18:09:11.876487 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.876459 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:09:11.928213 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928169 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls\") pod \"45d064ff-0a08-40f1-822b-8b73f7513956\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " Sep 16 18:09:11.928213 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928220 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location\") pod \"45d064ff-0a08-40f1-822b-8b73f7513956\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " Sep 16 18:09:11.928466 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928314 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5km6\" (UniqueName: \"kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6\") pod \"45d064ff-0a08-40f1-822b-8b73f7513956\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " Sep 16 18:09:11.928466 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928359 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config\") pod \"45d064ff-0a08-40f1-822b-8b73f7513956\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " Sep 16 18:09:11.928466 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928437 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert\") pod \"45d064ff-0a08-40f1-822b-8b73f7513956\" (UID: \"45d064ff-0a08-40f1-822b-8b73f7513956\") " Sep 16 18:09:11.928783 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928744 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45d064ff-0a08-40f1-822b-8b73f7513956" (UID: "45d064ff-0a08-40f1-822b-8b73f7513956"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:09:11.929014 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.928983 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45d064ff-0a08-40f1-822b-8b73f7513956" (UID: "45d064ff-0a08-40f1-822b-8b73f7513956"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:09:11.929079 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.929032 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45d064ff-0a08-40f1-822b-8b73f7513956" (UID: "45d064ff-0a08-40f1-822b-8b73f7513956"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:09:11.930841 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.930818 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45d064ff-0a08-40f1-822b-8b73f7513956" (UID: "45d064ff-0a08-40f1-822b-8b73f7513956"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:09:11.931047 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:11.931015 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6" (OuterVolumeSpecName: "kube-api-access-n5km6") pod "45d064ff-0a08-40f1-822b-8b73f7513956" (UID: "45d064ff-0a08-40f1-822b-8b73f7513956"). InnerVolumeSpecName "kube-api-access-n5km6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:09:12.029674 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.029640 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5km6\" (UniqueName: \"kubernetes.io/projected/45d064ff-0a08-40f1-822b-8b73f7513956-kube-api-access-n5km6\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:09:12.029674 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.029670 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:09:12.029674 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.029680 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45d064ff-0a08-40f1-822b-8b73f7513956-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:09:12.030017 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.029689 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45d064ff-0a08-40f1-822b-8b73f7513956-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:09:12.030017 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.029700 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d064ff-0a08-40f1-822b-8b73f7513956-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:09:12.377129 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.377037 2536 generic.go:356] "Generic (PLEG): container finished" podID="45d064ff-0a08-40f1-822b-8b73f7513956" containerID="7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426" exitCode=0 Sep 16 18:09:12.377129 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.377091 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" Sep 16 18:09:12.377129 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.377113 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerDied","Data":"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426"} Sep 16 18:09:12.377371 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.377183 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56" event={"ID":"45d064ff-0a08-40f1-822b-8b73f7513956","Type":"ContainerDied","Data":"31713e6db6fe66c650397cb375bda80bc28fc5180c0035980931d4b4a7151c31"} Sep 16 18:09:12.377371 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.377221 2536 scope.go:117] "RemoveContainer" containerID="b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306" Sep 16 18:09:12.386022 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.386001 2536 scope.go:117] "RemoveContainer" containerID="7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426" Sep 16 18:09:12.393847 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.393824 2536 scope.go:117] "RemoveContainer" containerID="7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6" Sep 16 18:09:12.401477 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.401452 2536 scope.go:117] "RemoveContainer" containerID="b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306" Sep 16 18:09:12.401766 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:09:12.401746 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306\": container with ID starting with b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306 not found: ID does not exist" containerID="b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306" Sep 16 18:09:12.401818 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.401777 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306"} err="failed to get container status \"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306\": rpc error: code = NotFound desc = could not find container \"b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306\": container with ID starting with b34caaa4863e5cb45131e68d0360fad3131f4c77a513a6e582b12bd58bb44306 not found: ID does not exist" Sep 16 18:09:12.401818 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.401794 2536 scope.go:117] "RemoveContainer" containerID="7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426" Sep 16 18:09:12.402119 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:09:12.402098 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426\": container with ID starting with 7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426 not found: ID does not exist" containerID="7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426" Sep 16 18:09:12.402182 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.402123 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426"} err="failed to get container status \"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426\": rpc error: code = NotFound desc = could not find container \"7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426\": container with ID starting with 7c57a3285e2bffdffb0e96b76e189064df14b8be7b85bfbee526854a7f47e426 not found: ID does not exist" Sep 16 18:09:12.402182 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.402138 2536 scope.go:117] "RemoveContainer" containerID="7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6" Sep 16 18:09:12.402399 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:09:12.402378 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6\": container with ID starting with 7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6 not found: ID does not exist" containerID="7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6" Sep 16 18:09:12.402451 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.402410 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6"} err="failed to get container status \"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6\": rpc error: code = NotFound desc = could not find container \"7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6\": container with ID starting with 7313f4a76413c770343545d2b8b82a960d1f16b6737fb587d15d62c67099e0a6 not found: ID does not exist" Sep 16 18:09:12.419984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.419951 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:09:12.421442 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.421419 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-22v56"] Sep 16 18:09:12.478422 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:12.478393 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" path="/var/lib/kubelet/pods/45d064ff-0a08-40f1-822b-8b73f7513956/volumes" Sep 16 18:09:19.895012 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.894974 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895309 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="storage-initializer" Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895321 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="storage-initializer" Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895335 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895341 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895354 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kube-rbac-proxy" Sep 16 18:09:19.895388 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895360 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kube-rbac-proxy" Sep 16 18:09:19.895586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895410 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kserve-container" Sep 16 18:09:19.895586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.895419 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d064ff-0a08-40f1-822b-8b73f7513956" containerName="kube-rbac-proxy" Sep 16 18:09:19.900122 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.900098 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:19.902956 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.902927 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\"" Sep 16 18:09:19.902956 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.902950 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\"" Sep 16 18:09:19.903398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.903379 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\"" Sep 16 18:09:19.903603 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.903532 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:09:19.904378 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.904361 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:09:19.904477 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.904385 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:09:19.913528 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.913504 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:09:19.999332 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.999297 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:19.999546 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.999352 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:19.999546 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.999379 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:19.999546 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:19.999448 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rbg\" (UniqueName: \"kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.100760 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.100728 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.100760 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.100765 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.101051 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.100904 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rbg\" (UniqueName: \"kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.101051 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.101034 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.101154 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:09:20.101136 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found Sep 16 18:09:20.101207 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:09:20.101197 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls podName:32ef2381-621e-4d26-bdbf-a45ffb67e87e nodeName:}" failed. No retries permitted until 2026-09-16 18:09:20.601177867 +0000 UTC m=+2668.544499448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls") pod "isvc-sklearn-s3-predictor-6957ddcb79-plzl6" (UID: "32ef2381-621e-4d26-bdbf-a45ffb67e87e") : secret "isvc-sklearn-s3-predictor-serving-cert" not found Sep 16 18:09:20.101274 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.101222 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.101450 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.101432 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.111663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.111644 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rbg\" (UniqueName: \"kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.605628 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.605586 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.608094 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.608063 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-plzl6\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:20.811423 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:20.811376 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:21.088207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:21.088182 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:09:21.090412 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:09:21.090385 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ef2381_621e_4d26_bdbf_a45ffb67e87e.slice/crio-a732c13735085ea282f7d9da530db4261eb8d747c5db3d99924b17ec5193809f WatchSource:0}: Error finding container a732c13735085ea282f7d9da530db4261eb8d747c5db3d99924b17ec5193809f: Status 404 returned error can't find the container with id a732c13735085ea282f7d9da530db4261eb8d747c5db3d99924b17ec5193809f Sep 16 18:09:21.419746 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:21.419654 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerStarted","Data":"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225"} Sep 16 18:09:21.419746 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:21.419714 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerStarted","Data":"a732c13735085ea282f7d9da530db4261eb8d747c5db3d99924b17ec5193809f"} Sep 16 18:09:23.429452 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:23.429419 2536 generic.go:356] "Generic (PLEG): container finished" podID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerID="bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225" exitCode=0 Sep 16 18:09:23.429831 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:23.429481 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerDied","Data":"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225"} Sep 16 18:09:24.436541 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:24.436508 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerStarted","Data":"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35"} Sep 16 18:09:24.437034 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:24.436569 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerStarted","Data":"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075"} Sep 16 18:09:24.437034 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:24.436807 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:25.441234 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:25.441194 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:25.442532 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:25.442502 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:09:26.445750 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:26.445704 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:09:31.450717 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:31.450685 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:09:31.451372 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:31.451341 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:09:31.475823 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:31.475766 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podStartSLOduration=10.886762486 podStartE2EDuration="12.475748903s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.588986417s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:24.467412585 +0000 UTC m=+2672.410734223" watchObservedRunningTime="2026-09-16 18:09:31.475748903 +0000 UTC m=+2679.419070505" Sep 16 18:09:41.451985 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:41.451946 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:09:51.451556 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:51.451516 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:09:52.629793 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:52.629767 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:09:52.637125 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:09:52.637103 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:10:01.451684 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:01.451645 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:10:11.451474 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:11.451431 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:10:21.451586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:21.451547 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:10:31.451764 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:31.451725 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:10:41.452697 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:41.452665 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:10:50.068893 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.068843 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:10:50.069498 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.069125 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" containerID="cri-o://1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075" gracePeriod=30 Sep 16 18:10:50.069498 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.069163 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kube-rbac-proxy" containerID="cri-o://15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35" gracePeriod=30 Sep 16 18:10:50.423289 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.423201 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:10:50.432781 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.432749 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.439787 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.439756 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\"" Sep 16 18:10:50.439905 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.439796 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\"" Sep 16 18:10:50.439905 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.439771 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:10:50.461852 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.461820 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:10:50.469083 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.469054 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.469232 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.469099 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747n6\" (UniqueName: \"kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.469232 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.469169 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.469232 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.469209 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.469341 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.469240 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.569722 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.569684 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747n6\" (UniqueName: \"kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.569960 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.569738 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.569960 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.569770 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.569960 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.569820 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.569960 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.569953 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.570454 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.570412 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.570657 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.570630 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.570970 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.570945 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.572558 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.572536 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.598859 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.598823 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-747n6\" (UniqueName: \"kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.743061 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.743025 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:50.818875 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.818835 2536 generic.go:356] "Generic (PLEG): container finished" podID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerID="15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35" exitCode=2 Sep 16 18:10:50.819517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:50.819491 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerDied","Data":"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35"} Sep 16 18:10:51.026299 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:51.026271 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:10:51.028909 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:10:51.028882 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4270cf48_b923_48d9_a0f7_b84b50a5aa4d.slice/crio-f0483f98ba027e1c2476969e5671271b66e7f828a68a14c594067c42d638a827 WatchSource:0}: Error finding container f0483f98ba027e1c2476969e5671271b66e7f828a68a14c594067c42d638a827: Status 404 returned error can't find the container with id f0483f98ba027e1c2476969e5671271b66e7f828a68a14c594067c42d638a827 Sep 16 18:10:51.446734 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:51.446623 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused" Sep 16 18:10:51.451491 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:51.451461 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused" Sep 16 18:10:51.825994 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:51.825957 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerStarted","Data":"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3"} Sep 16 18:10:51.826162 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:51.826043 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerStarted","Data":"f0483f98ba027e1c2476969e5671271b66e7f828a68a14c594067c42d638a827"} Sep 16 18:10:52.831813 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:52.831781 2536 generic.go:356] "Generic (PLEG): container finished" podID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerID="3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3" exitCode=0 Sep 16 18:10:52.834103 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:52.831834 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerDied","Data":"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3"} Sep 16 18:10:53.838933 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.838871 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerStarted","Data":"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807"} Sep 16 18:10:53.839362 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.838953 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerStarted","Data":"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072"} Sep 16 18:10:53.839362 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.839088 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:53.839362 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.839138 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:53.840290 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.840268 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:10:53.862139 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:53.862085 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podStartSLOduration=2.732634596 podStartE2EDuration="3.862070603s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.129436007s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:10:53.86105996 +0000 UTC m=+2761.804381588" watchObservedRunningTime="2026-09-16 18:10:53.862070603 +0000 UTC m=+2761.805392207" Sep 16 18:10:54.843992 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:54.843952 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:10:55.118590 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.118566 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:10:55.212932 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.212873 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95rbg\" (UniqueName: \"kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg\") pod \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " Sep 16 18:10:55.213114 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.212975 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config\") pod \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " Sep 16 18:10:55.213114 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.213049 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location\") pod \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " Sep 16 18:10:55.213114 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.213097 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") pod \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\" (UID: \"32ef2381-621e-4d26-bdbf-a45ffb67e87e\") " Sep 16 18:10:55.213372 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.213339 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32ef2381-621e-4d26-bdbf-a45ffb67e87e" (UID: "32ef2381-621e-4d26-bdbf-a45ffb67e87e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:10:55.213439 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.213376 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32ef2381-621e-4d26-bdbf-a45ffb67e87e" (UID: "32ef2381-621e-4d26-bdbf-a45ffb67e87e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:10:55.215299 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.215272 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg" (OuterVolumeSpecName: "kube-api-access-95rbg") pod "32ef2381-621e-4d26-bdbf-a45ffb67e87e" (UID: "32ef2381-621e-4d26-bdbf-a45ffb67e87e"). InnerVolumeSpecName "kube-api-access-95rbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:10:55.215363 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.215307 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32ef2381-621e-4d26-bdbf-a45ffb67e87e" (UID: "32ef2381-621e-4d26-bdbf-a45ffb67e87e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:10:55.314040 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.314004 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:10:55.314040 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.314033 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:10:55.314040 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.314043 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ef2381-621e-4d26-bdbf-a45ffb67e87e-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:10:55.314040 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.314052 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rbg\" (UniqueName: \"kubernetes.io/projected/32ef2381-621e-4d26-bdbf-a45ffb67e87e-kube-api-access-95rbg\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:10:55.850778 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.850745 2536 generic.go:356] "Generic (PLEG): container finished" podID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerID="1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075" exitCode=0 Sep 16 18:10:55.851207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.850796 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" Sep 16 18:10:55.851207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.850810 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerDied","Data":"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075"} Sep 16 18:10:55.851207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.850868 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6" event={"ID":"32ef2381-621e-4d26-bdbf-a45ffb67e87e","Type":"ContainerDied","Data":"a732c13735085ea282f7d9da530db4261eb8d747c5db3d99924b17ec5193809f"} Sep 16 18:10:55.851207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.850897 2536 scope.go:117] "RemoveContainer" containerID="15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35" Sep 16 18:10:55.859953 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.859933 2536 scope.go:117] "RemoveContainer" containerID="1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075" Sep 16 18:10:55.867800 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.867778 2536 scope.go:117] "RemoveContainer" containerID="bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225" Sep 16 18:10:55.875270 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.875243 2536 scope.go:117] "RemoveContainer" containerID="15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35" Sep 16 18:10:55.875568 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:10:55.875548 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35\": container with ID starting with 15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35 not found: ID does not exist" containerID="15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35" Sep 16 18:10:55.875622 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.875578 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35"} err="failed to get container status \"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35\": rpc error: code = NotFound desc = could not find container \"15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35\": container with ID starting with 15db7135721386b44891e28b4fc1d2dfaec6b717e020dbaf9d083b66147d4e35 not found: ID does not exist" Sep 16 18:10:55.875622 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.875598 2536 scope.go:117] "RemoveContainer" containerID="1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075" Sep 16 18:10:55.875879 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:10:55.875860 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075\": container with ID starting with 1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075 not found: ID does not exist" containerID="1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075" Sep 16 18:10:55.875940 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.875885 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075"} err="failed to get container status \"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075\": rpc error: code = NotFound desc = could not find container \"1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075\": container with ID starting with 1c8b0d73b480364ac5466d6a92709e4e1785220933ccc6c76393afff0e063075 not found: ID does not exist" Sep 16 18:10:55.875940 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.875902 2536 scope.go:117] "RemoveContainer" containerID="bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225" Sep 16 18:10:55.876376 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:10:55.876354 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225\": container with ID starting with bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225 not found: ID does not exist" containerID="bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225" Sep 16 18:10:55.876428 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.876384 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225"} err="failed to get container status \"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225\": rpc error: code = NotFound desc = could not find container \"bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225\": container with ID starting with bbe8714c6461d68ec9ba2284e87b79950d9e9069d92bfda032dd9e44d6e08225 not found: ID does not exist" Sep 16 18:10:55.885590 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.885562 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:10:55.894009 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:55.893974 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-plzl6"] Sep 16 18:10:56.477969 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:56.477935 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" path="/var/lib/kubelet/pods/32ef2381-621e-4d26-bdbf-a45ffb67e87e/volumes" Sep 16 18:10:59.848786 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:59.848760 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:10:59.849284 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:10:59.849253 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:09.849334 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:09.849295 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:19.849846 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:19.849759 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:29.849782 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:29.849741 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:39.850214 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:39.850166 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:49.849499 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:49.849458 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:11:59.849352 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:11:59.849309 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused" Sep 16 18:12:09.850078 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:09.850042 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:12:10.235027 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:10.234993 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:12:10.235258 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:10.235232 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kube-rbac-proxy" containerID="cri-o://4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807" gracePeriod=30 Sep 16 18:12:10.235353 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:10.235210 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" containerID="cri-o://73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072" gracePeriod=30 Sep 16 18:12:11.182046 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.182009 2536 generic.go:356] "Generic (PLEG): container finished" podID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerID="4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807" exitCode=2 Sep 16 18:12:11.182505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.182087 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerDied","Data":"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807"} Sep 16 18:12:11.365275 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365239 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:11.365618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365603 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="storage-initializer" Sep 16 18:12:11.365618 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365619 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="storage-initializer" Sep 16 18:12:11.365721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365634 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" Sep 16 18:12:11.365721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365640 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" Sep 16 18:12:11.365721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365646 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kube-rbac-proxy" Sep 16 18:12:11.365721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365652 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kube-rbac-proxy" Sep 16 18:12:11.365721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365717 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kserve-container" Sep 16 18:12:11.365882 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.365725 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ef2381-621e-4d26-bdbf-a45ffb67e87e" containerName="kube-rbac-proxy" Sep 16 18:12:11.368904 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.368888 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.374358 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.374334 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\"" Sep 16 18:12:11.374474 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.374384 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\"" Sep 16 18:12:11.386155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.386125 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:11.436476 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.436372 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.436643 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.436488 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49hs\" (UniqueName: \"kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.436643 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.436535 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.436643 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.436554 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537193 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537151 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537193 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537192 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537329 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537426 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49hs\" (UniqueName: \"kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537722 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.537911 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.537887 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.539738 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.539718 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.547628 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.547602 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49hs\" (UniqueName: \"kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.679059 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.679016 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:11.959772 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:11.959627 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:11.962468 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:12:11.962440 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31950869_7b4c_4343_bb16_035d200d6bc9.slice/crio-5e0a9dc73b41e1884cf413bdc1e4f2c45f0055f501a0070d722e650bf6724450 WatchSource:0}: Error finding container 5e0a9dc73b41e1884cf413bdc1e4f2c45f0055f501a0070d722e650bf6724450: Status 404 returned error can't find the container with id 5e0a9dc73b41e1884cf413bdc1e4f2c45f0055f501a0070d722e650bf6724450 Sep 16 18:12:12.188734 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:12.188701 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerStarted","Data":"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804"} Sep 16 18:12:12.189218 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:12.188756 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerStarted","Data":"5e0a9dc73b41e1884cf413bdc1e4f2c45f0055f501a0070d722e650bf6724450"} Sep 16 18:12:14.845126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:14.845076 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused" Sep 16 18:12:15.379452 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.379421 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:12:15.469225 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469196 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-747n6\" (UniqueName: \"kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6\") pod \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " Sep 16 18:12:15.469392 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469257 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert\") pod \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " Sep 16 18:12:15.469392 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469276 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config\") pod \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " Sep 16 18:12:15.469392 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469303 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls\") pod \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " Sep 16 18:12:15.469392 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469334 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location\") pod \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\" (UID: \"4270cf48-b923-48d9-a0f7-b84b50a5aa4d\") " Sep 16 18:12:15.469751 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469722 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4270cf48-b923-48d9-a0f7-b84b50a5aa4d" (UID: "4270cf48-b923-48d9-a0f7-b84b50a5aa4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:12:15.469751 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469740 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4270cf48-b923-48d9-a0f7-b84b50a5aa4d" (UID: "4270cf48-b923-48d9-a0f7-b84b50a5aa4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:12:15.469890 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.469718 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4270cf48-b923-48d9-a0f7-b84b50a5aa4d" (UID: "4270cf48-b923-48d9-a0f7-b84b50a5aa4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:12:15.471459 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.471432 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4270cf48-b923-48d9-a0f7-b84b50a5aa4d" (UID: "4270cf48-b923-48d9-a0f7-b84b50a5aa4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:12:15.471575 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.471486 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6" (OuterVolumeSpecName: "kube-api-access-747n6") pod "4270cf48-b923-48d9-a0f7-b84b50a5aa4d" (UID: "4270cf48-b923-48d9-a0f7-b84b50a5aa4d"). InnerVolumeSpecName "kube-api-access-747n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:12:15.570225 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.570166 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-747n6\" (UniqueName: \"kubernetes.io/projected/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-api-access-747n6\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:15.570225 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.570218 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:15.570225 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.570232 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:15.570483 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.570244 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:15.570483 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:15.570257 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4270cf48-b923-48d9-a0f7-b84b50a5aa4d-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:16.209478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.209445 2536 generic.go:356] "Generic (PLEG): container finished" podID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerID="73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072" exitCode=0 Sep 16 18:12:16.209984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.209505 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerDied","Data":"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072"} Sep 16 18:12:16.209984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.209507 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" Sep 16 18:12:16.209984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.209552 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2" event={"ID":"4270cf48-b923-48d9-a0f7-b84b50a5aa4d","Type":"ContainerDied","Data":"f0483f98ba027e1c2476969e5671271b66e7f828a68a14c594067c42d638a827"} Sep 16 18:12:16.209984 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.209577 2536 scope.go:117] "RemoveContainer" containerID="4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807" Sep 16 18:12:16.218288 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.218269 2536 scope.go:117] "RemoveContainer" containerID="73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072" Sep 16 18:12:16.225655 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.225637 2536 scope.go:117] "RemoveContainer" containerID="3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3" Sep 16 18:12:16.232889 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.232873 2536 scope.go:117] "RemoveContainer" containerID="4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807" Sep 16 18:12:16.233196 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:12:16.233174 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807\": container with ID starting with 4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807 not found: ID does not exist" containerID="4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807" Sep 16 18:12:16.233239 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.233205 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807"} err="failed to get container status \"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807\": rpc error: code = NotFound desc = could not find container \"4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807\": container with ID starting with 4e25d2cb8ee611a2170f521a8e48fbe8b4e4d91e574739a01391bc76cf56a807 not found: ID does not exist" Sep 16 18:12:16.233239 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.233222 2536 scope.go:117] "RemoveContainer" containerID="73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072" Sep 16 18:12:16.233499 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:12:16.233481 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072\": container with ID starting with 73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072 not found: ID does not exist" containerID="73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072" Sep 16 18:12:16.233540 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.233505 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072"} err="failed to get container status \"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072\": rpc error: code = NotFound desc = could not find container \"73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072\": container with ID starting with 73ab26150538785b13335cd81886abf125c6d75606bc9f0a7398600271e4d072 not found: ID does not exist" Sep 16 18:12:16.233540 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.233521 2536 scope.go:117] "RemoveContainer" containerID="3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3" Sep 16 18:12:16.233741 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:12:16.233726 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3\": container with ID starting with 3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3 not found: ID does not exist" containerID="3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3" Sep 16 18:12:16.233789 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.233745 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3"} err="failed to get container status \"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3\": rpc error: code = NotFound desc = could not find container \"3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3\": container with ID starting with 3f0fd911a8baedc411886215b8e9c584e2e2db7d87bac6b2aa8a8f483ecf04f3 not found: ID does not exist" Sep 16 18:12:16.239514 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.239491 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:12:16.257082 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.257053 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-nkbp2"] Sep 16 18:12:16.478113 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:16.478086 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" path="/var/lib/kubelet/pods/4270cf48-b923-48d9-a0f7-b84b50a5aa4d/volumes" Sep 16 18:12:19.226062 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:19.226033 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/0.log" Sep 16 18:12:19.226415 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:19.226078 2536 generic.go:356] "Generic (PLEG): container finished" podID="31950869-7b4c-4343-bb16-035d200d6bc9" containerID="3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804" exitCode=1 Sep 16 18:12:19.226415 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:19.226133 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerDied","Data":"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804"} Sep 16 18:12:20.233292 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:20.233263 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/0.log" Sep 16 18:12:20.233703 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:20.233381 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerStarted","Data":"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68"} Sep 16 18:12:21.313748 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:21.313711 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:21.314235 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:21.314012 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" containerID="cri-o://4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68" gracePeriod=30 Sep 16 18:12:22.477762 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.477735 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478050 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="storage-initializer" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478062 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="storage-initializer" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478075 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478081 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478089 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kube-rbac-proxy" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478095 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kube-rbac-proxy" Sep 16 18:12:22.478155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478155 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kube-rbac-proxy" Sep 16 18:12:22.478474 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.478164 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4270cf48-b923-48d9-a0f7-b84b50a5aa4d" containerName="kserve-container" Sep 16 18:12:22.481301 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.481282 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.484126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.484105 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\"" Sep 16 18:12:22.484126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.484118 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:12:22.484358 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.484346 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\"" Sep 16 18:12:22.496063 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.496031 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:12:22.630287 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.630248 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.630287 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.630299 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqd4\" (UniqueName: \"kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.630520 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.630320 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.630520 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.630377 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.630520 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.630454 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731173 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731084 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731173 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731134 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqd4\" (UniqueName: \"kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731413 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731249 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731413 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731303 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731413 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731342 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731659 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731638 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731792 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731774 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.731905 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.731885 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.733797 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.733771 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.741288 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.741268 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqd4\" (UniqueName: \"kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:22.792143 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:22.792106 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:23.070875 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.070844 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:12:23.078985 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:12:23.078951 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b085d2_fca6_4284_ba23_31498e7f2b66.slice/crio-afde6851ea453e6d9a34fc66071d26d5dfa444aa43c6ba01bc295e88fa17aa36 WatchSource:0}: Error finding container afde6851ea453e6d9a34fc66071d26d5dfa444aa43c6ba01bc295e88fa17aa36: Status 404 returned error can't find the container with id afde6851ea453e6d9a34fc66071d26d5dfa444aa43c6ba01bc295e88fa17aa36 Sep 16 18:12:23.165410 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.165386 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/1.log" Sep 16 18:12:23.165771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.165752 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/0.log" Sep 16 18:12:23.168234 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.168213 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:23.235293 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235261 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls\") pod \"31950869-7b4c-4343-bb16-035d200d6bc9\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " Sep 16 18:12:23.235496 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235313 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w49hs\" (UniqueName: \"kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs\") pod \"31950869-7b4c-4343-bb16-035d200d6bc9\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " Sep 16 18:12:23.235496 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235371 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config\") pod \"31950869-7b4c-4343-bb16-035d200d6bc9\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " Sep 16 18:12:23.235496 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235399 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location\") pod \"31950869-7b4c-4343-bb16-035d200d6bc9\" (UID: \"31950869-7b4c-4343-bb16-035d200d6bc9\") " Sep 16 18:12:23.235719 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235696 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31950869-7b4c-4343-bb16-035d200d6bc9" (UID: "31950869-7b4c-4343-bb16-035d200d6bc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:12:23.235814 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.235787 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31950869-7b4c-4343-bb16-035d200d6bc9" (UID: "31950869-7b4c-4343-bb16-035d200d6bc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:12:23.237490 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.237451 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs" (OuterVolumeSpecName: "kube-api-access-w49hs") pod "31950869-7b4c-4343-bb16-035d200d6bc9" (UID: "31950869-7b4c-4343-bb16-035d200d6bc9"). InnerVolumeSpecName "kube-api-access-w49hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:12:23.237587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.237503 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31950869-7b4c-4343-bb16-035d200d6bc9" (UID: "31950869-7b4c-4343-bb16-035d200d6bc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:12:23.246962 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.246942 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/1.log" Sep 16 18:12:23.247437 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247420 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25_31950869-7b4c-4343-bb16-035d200d6bc9/storage-initializer/0.log" Sep 16 18:12:23.247530 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247459 2536 generic.go:356] "Generic (PLEG): container finished" podID="31950869-7b4c-4343-bb16-035d200d6bc9" containerID="4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68" exitCode=1 Sep 16 18:12:23.247530 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247515 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" Sep 16 18:12:23.247631 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247556 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerDied","Data":"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68"} Sep 16 18:12:23.247696 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247632 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25" event={"ID":"31950869-7b4c-4343-bb16-035d200d6bc9","Type":"ContainerDied","Data":"5e0a9dc73b41e1884cf413bdc1e4f2c45f0055f501a0070d722e650bf6724450"} Sep 16 18:12:23.247696 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.247673 2536 scope.go:117] "RemoveContainer" containerID="4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68" Sep 16 18:12:23.249417 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.249387 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerStarted","Data":"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b"} Sep 16 18:12:23.249533 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.249453 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerStarted","Data":"afde6851ea453e6d9a34fc66071d26d5dfa444aa43c6ba01bc295e88fa17aa36"} Sep 16 18:12:23.257344 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.257326 2536 scope.go:117] "RemoveContainer" containerID="3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804" Sep 16 18:12:23.265031 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.265005 2536 scope.go:117] "RemoveContainer" containerID="4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68" Sep 16 18:12:23.265329 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:12:23.265308 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68\": container with ID starting with 4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68 not found: ID does not exist" containerID="4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68" Sep 16 18:12:23.265401 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.265337 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68"} err="failed to get container status \"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68\": rpc error: code = NotFound desc = could not find container \"4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68\": container with ID starting with 4406655197ba165f8bee65b5024e733f3f435a0306e32d13b8eb8040e5236a68 not found: ID does not exist" Sep 16 18:12:23.265401 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.265358 2536 scope.go:117] "RemoveContainer" containerID="3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804" Sep 16 18:12:23.265641 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:12:23.265618 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804\": container with ID starting with 3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804 not found: ID does not exist" containerID="3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804" Sep 16 18:12:23.265713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.265647 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804"} err="failed to get container status \"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804\": rpc error: code = NotFound desc = could not find container \"3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804\": container with ID starting with 3d4ff27d122825cefd7cf7e737b103d1cdf275876b82bc90d8311fa195731804 not found: ID does not exist" Sep 16 18:12:23.310782 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.310739 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:23.315613 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.315583 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-slm25"] Sep 16 18:12:23.336596 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.336559 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31950869-7b4c-4343-bb16-035d200d6bc9-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:23.336596 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.336599 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49hs\" (UniqueName: \"kubernetes.io/projected/31950869-7b4c-4343-bb16-035d200d6bc9-kube-api-access-w49hs\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:23.336811 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.336616 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31950869-7b4c-4343-bb16-035d200d6bc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:23.336811 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:23.336632 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31950869-7b4c-4343-bb16-035d200d6bc9-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:12:24.477784 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:24.477749 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" path="/var/lib/kubelet/pods/31950869-7b4c-4343-bb16-035d200d6bc9/volumes" Sep 16 18:12:25.260998 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:25.260965 2536 generic.go:356] "Generic (PLEG): container finished" podID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerID="e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b" exitCode=0 Sep 16 18:12:25.261161 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:25.261037 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerDied","Data":"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b"} Sep 16 18:12:26.268090 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:26.268053 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerStarted","Data":"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d"} Sep 16 18:12:26.268451 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:26.268119 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerStarted","Data":"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339"} Sep 16 18:12:26.268451 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:26.268313 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:26.291496 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:26.291450 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podStartSLOduration=3.120904631 podStartE2EDuration="4.291434043s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.170529412s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:12:26.291143891 +0000 UTC m=+2854.234465539" watchObservedRunningTime="2026-09-16 18:12:26.291434043 +0000 UTC m=+2854.234755646" Sep 16 18:12:27.272811 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:27.272780 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:27.274196 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:27.274172 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:12:28.277038 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:28.277001 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:12:33.281932 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:33.281888 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:12:33.282493 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:33.282465 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:12:43.282784 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:43.282743 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:12:53.282860 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.282820 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:12:53.303325 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303286 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:12:53.303648 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303634 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.303696 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303650 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.303743 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303716 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.303743 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303724 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.303818 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303785 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.303818 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.303791 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="31950869-7b4c-4343-bb16-035d200d6bc9" containerName="storage-initializer" Sep 16 18:12:53.308440 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.308417 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.311220 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.311199 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\"" Sep 16 18:12:53.311372 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.311229 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:12:53.311488 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.311472 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\"" Sep 16 18:12:53.318464 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.318436 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:12:53.399747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.399709 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.399747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.399748 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.400021 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.399789 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcsl\" (UniqueName: \"kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.400021 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.399848 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.400021 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.399910 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501088 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501049 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501088 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501090 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501346 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501128 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcsl\" (UniqueName: \"kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501346 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501161 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501346 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501191 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501579 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501555 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501842 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501819 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.501943 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.501856 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.503614 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.503597 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.511049 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.511029 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcsl\" (UniqueName: \"kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lf4pf\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.620116 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.620019 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:12:53.898048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:53.898015 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:12:53.899754 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:12:53.899729 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b26290_92fd_4b0d_9d6f_d3db1a7f4daa.slice/crio-30f6397094d8a399e753a0ce2ac67dd2aabd3cf3f30ae8ab4ef4ee7697c96695 WatchSource:0}: Error finding container 30f6397094d8a399e753a0ce2ac67dd2aabd3cf3f30ae8ab4ef4ee7697c96695: Status 404 returned error can't find the container with id 30f6397094d8a399e753a0ce2ac67dd2aabd3cf3f30ae8ab4ef4ee7697c96695 Sep 16 18:12:54.399368 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:54.399334 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerStarted","Data":"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e"} Sep 16 18:12:54.399762 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:54.399393 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerStarted","Data":"30f6397094d8a399e753a0ce2ac67dd2aabd3cf3f30ae8ab4ef4ee7697c96695"} Sep 16 18:12:59.427264 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:59.427227 2536 generic.go:356] "Generic (PLEG): container finished" podID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerID="d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e" exitCode=0 Sep 16 18:12:59.427650 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:59.427292 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerDied","Data":"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e"} Sep 16 18:12:59.428531 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:12:59.428514 2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:13:03.282958 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:03.282895 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:13:11.499720 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:11.499679 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerStarted","Data":"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1"} Sep 16 18:13:11.500150 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:11.499796 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerStarted","Data":"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79"} Sep 16 18:13:11.500150 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:11.500057 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:13:11.523222 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:11.523151 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podStartSLOduration=2.082376736 podStartE2EDuration="18.523134203s" totalImagesPullingTime="11.734927289s" totalInitContainerRuntime="4.705830178s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:13:11.522731902 +0000 UTC m=+2899.466053542" watchObservedRunningTime="2026-09-16 18:13:11.523134203 +0000 UTC m=+2899.466455842" Sep 16 18:13:12.504501 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:12.504464 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:13:12.505863 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:12.505836 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:13.283124 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:13.283085 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:13:13.508619 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:13.508575 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:18.513297 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:18.513262 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:13:18.514048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:18.514014 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:23.282807 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:23.282767 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:13:28.514567 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:28.514526 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:33.282543 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:33.282498 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:13:38.514725 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:38.514686 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:43.283675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:43.283638 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:13:48.514500 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:48.514461 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused" Sep 16 18:13:52.484541 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:52.484508 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:13:52.485101 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:52.484780 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" containerID="cri-o://41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339" gracePeriod=30 Sep 16 18:13:52.485101 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:52.484794 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kube-rbac-proxy" containerID="cri-o://1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d" gracePeriod=30 Sep 16 18:13:52.686274 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:52.686231 2536 generic.go:356] "Generic (PLEG): container finished" podID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerID="1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d" exitCode=2 Sep 16 18:13:52.686439 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:52.686302 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerDied","Data":"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d"} Sep 16 18:13:53.278198 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:53.278154 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused" Sep 16 18:13:53.282501 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:53.282472 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused" Sep 16 18:13:57.328947 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.328909 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:13:57.442648 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442557 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvqd4\" (UniqueName: \"kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4\") pod \"51b085d2-fca6-4284-ba23-31498e7f2b66\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " Sep 16 18:13:57.442648 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442604 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location\") pod \"51b085d2-fca6-4284-ba23-31498e7f2b66\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " Sep 16 18:13:57.442648 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442627 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert\") pod \"51b085d2-fca6-4284-ba23-31498e7f2b66\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " Sep 16 18:13:57.442648 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442644 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls\") pod \"51b085d2-fca6-4284-ba23-31498e7f2b66\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " Sep 16 18:13:57.443032 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442773 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config\") pod \"51b085d2-fca6-4284-ba23-31498e7f2b66\" (UID: \"51b085d2-fca6-4284-ba23-31498e7f2b66\") " Sep 16 18:13:57.443032 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442955 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51b085d2-fca6-4284-ba23-31498e7f2b66" (UID: "51b085d2-fca6-4284-ba23-31498e7f2b66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:13:57.443032 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.442989 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51b085d2-fca6-4284-ba23-31498e7f2b66" (UID: "51b085d2-fca6-4284-ba23-31498e7f2b66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:13:57.443168 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.443147 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51b085d2-fca6-4284-ba23-31498e7f2b66" (UID: "51b085d2-fca6-4284-ba23-31498e7f2b66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:13:57.443206 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.443173 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b085d2-fca6-4284-ba23-31498e7f2b66-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:13:57.443206 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.443192 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:13:57.445349 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.445324 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51b085d2-fca6-4284-ba23-31498e7f2b66" (UID: "51b085d2-fca6-4284-ba23-31498e7f2b66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:13:57.445446 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.445327 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4" (OuterVolumeSpecName: "kube-api-access-zvqd4") pod "51b085d2-fca6-4284-ba23-31498e7f2b66" (UID: "51b085d2-fca6-4284-ba23-31498e7f2b66"). InnerVolumeSpecName "kube-api-access-zvqd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:13:57.543848 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.543808 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b085d2-fca6-4284-ba23-31498e7f2b66-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:13:57.543848 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.543842 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:13:57.543848 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.543855 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqd4\" (UniqueName: \"kubernetes.io/projected/51b085d2-fca6-4284-ba23-31498e7f2b66-kube-api-access-zvqd4\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:13:57.710694 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.710605 2536 generic.go:356] "Generic (PLEG): container finished" podID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerID="41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339" exitCode=0 Sep 16 18:13:57.710694 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.710670 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" Sep 16 18:13:57.710883 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.710688 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerDied","Data":"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339"} Sep 16 18:13:57.710883 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.710753 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr" event={"ID":"51b085d2-fca6-4284-ba23-31498e7f2b66","Type":"ContainerDied","Data":"afde6851ea453e6d9a34fc66071d26d5dfa444aa43c6ba01bc295e88fa17aa36"} Sep 16 18:13:57.710883 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.710780 2536 scope.go:117] "RemoveContainer" containerID="1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d" Sep 16 18:13:57.719389 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.719365 2536 scope.go:117] "RemoveContainer" containerID="41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339" Sep 16 18:13:57.726811 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.726794 2536 scope.go:117] "RemoveContainer" containerID="e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b" Sep 16 18:13:57.734899 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.734881 2536 scope.go:117] "RemoveContainer" containerID="1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d" Sep 16 18:13:57.735267 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:13:57.735206 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d\": container with ID starting with 1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d not found: ID does not exist" containerID="1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d" Sep 16 18:13:57.735267 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.735240 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d"} err="failed to get container status \"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d\": rpc error: code = NotFound desc = could not find container \"1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d\": container with ID starting with 1e36c45be2faba4eecd25c1a9d2366ca2e65a66bc8783d96b659b1a93d992a5d not found: ID does not exist" Sep 16 18:13:57.735267 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.735264 2536 scope.go:117] "RemoveContainer" containerID="41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339" Sep 16 18:13:57.735591 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:13:57.735569 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339\": container with ID starting with 41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339 not found: ID does not exist" containerID="41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339" Sep 16 18:13:57.735632 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.735602 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339"} err="failed to get container status \"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339\": rpc error: code = NotFound desc = could not find container \"41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339\": container with ID starting with 41f9b9d50d51665b874b50ca65dcda26ad17aa933980e1919a2354cfd68f2339 not found: ID does not exist" Sep 16 18:13:57.735632 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.735623 2536 scope.go:117] "RemoveContainer" containerID="e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b" Sep 16 18:13:57.735906 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:13:57.735889 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b\": container with ID starting with e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b not found: ID does not exist" containerID="e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b" Sep 16 18:13:57.735989 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.735911 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b"} err="failed to get container status \"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b\": rpc error: code = NotFound desc = could not find container \"e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b\": container with ID starting with e48ad6e58fa09c3710d53d15187e5bab8dc1dbd05e4475eed37734caed418f2b not found: ID does not exist" Sep 16 18:13:57.736145 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.736128 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:13:57.745800 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:57.745777 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-c6qtr"] Sep 16 18:13:58.477702 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:58.477670 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" path="/var/lib/kubelet/pods/51b085d2-fca6-4284-ba23-31498e7f2b66/volumes" Sep 16 18:13:58.515365 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:13:58.515335 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:14:04.424600 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:04.424565 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:14:04.425116 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:04.424797 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" containerID="cri-o://8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79" gracePeriod=30 Sep 16 18:14:04.425116 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:04.424862 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kube-rbac-proxy" containerID="cri-o://50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1" gracePeriod=30 Sep 16 18:14:04.753759 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:04.753729 2536 generic.go:356] "Generic (PLEG): container finished" podID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerID="50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1" exitCode=2 Sep 16 18:14:04.753973 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:04.753792 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerDied","Data":"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1"} Sep 16 18:14:07.171182 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.171158 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:14:07.223643 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.223610 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert\") pod \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " Sep 16 18:14:07.223824 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.223652 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location\") pod \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " Sep 16 18:14:07.223824 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.223723 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls\") pod \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " Sep 16 18:14:07.223824 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.223745 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config\") pod \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " Sep 16 18:14:07.223824 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.223767 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xcsl\" (UniqueName: \"kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl\") pod \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\" (UID: \"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa\") " Sep 16 18:14:07.224115 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.224090 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" (UID: "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:14:07.224164 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.224119 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" (UID: "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:14:07.225948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.225868 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl" (OuterVolumeSpecName: "kube-api-access-9xcsl") pod "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" (UID: "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa"). InnerVolumeSpecName "kube-api-access-9xcsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:14:07.225948 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.225911 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" (UID: "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:14:07.232445 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.232424 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" (UID: "36b26290-92fd-4b0d-9d6f-d3db1a7f4daa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:14:07.325257 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.325178 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:14:07.325257 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.325207 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:14:07.325257 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.325216 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xcsl\" (UniqueName: \"kubernetes.io/projected/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kube-api-access-9xcsl\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:14:07.325257 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.325227 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:14:07.325257 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.325235 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:14:07.769848 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.769807 2536 generic.go:356] "Generic (PLEG): container finished" podID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerID="8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79" exitCode=0 Sep 16 18:14:07.770138 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.769860 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerDied","Data":"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79"} Sep 16 18:14:07.770138 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.769884 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" Sep 16 18:14:07.770138 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.769944 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf" event={"ID":"36b26290-92fd-4b0d-9d6f-d3db1a7f4daa","Type":"ContainerDied","Data":"30f6397094d8a399e753a0ce2ac67dd2aabd3cf3f30ae8ab4ef4ee7697c96695"} Sep 16 18:14:07.770138 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.769971 2536 scope.go:117] "RemoveContainer" containerID="50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1" Sep 16 18:14:07.779039 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.779013 2536 scope.go:117] "RemoveContainer" containerID="8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79" Sep 16 18:14:07.786615 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.786594 2536 scope.go:117] "RemoveContainer" containerID="d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e" Sep 16 18:14:07.794040 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794012 2536 scope.go:117] "RemoveContainer" containerID="50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1" Sep 16 18:14:07.794320 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:14:07.794303 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1\": container with ID starting with 50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1 not found: ID does not exist" containerID="50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1" Sep 16 18:14:07.794400 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794333 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1"} err="failed to get container status \"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1\": rpc error: code = NotFound desc = could not find container \"50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1\": container with ID starting with 50e796dd0cb4e1777483186e6f382b02a284edafab2cdb702d1bc2a3d9e051f1 not found: ID does not exist" Sep 16 18:14:07.794400 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794359 2536 scope.go:117] "RemoveContainer" containerID="8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79" Sep 16 18:14:07.794635 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:14:07.794617 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79\": container with ID starting with 8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79 not found: ID does not exist" containerID="8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79" Sep 16 18:14:07.794677 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794641 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79"} err="failed to get container status \"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79\": rpc error: code = NotFound desc = could not find container \"8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79\": container with ID starting with 8730b491144a9b71bb041eb4a9ac6815e868376b40526cb3815120af7d4f2f79 not found: ID does not exist" Sep 16 18:14:07.794677 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794655 2536 scope.go:117] "RemoveContainer" containerID="d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e" Sep 16 18:14:07.794950 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:14:07.794910 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e\": container with ID starting with d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e not found: ID does not exist" containerID="d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e" Sep 16 18:14:07.795012 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.794955 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e"} err="failed to get container status \"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e\": rpc error: code = NotFound desc = could not find container \"d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e\": container with ID starting with d64d8ee1042d5616d6c3e80afda9b716198c4955ed7d8dea0b54dd8b099ae68e not found: ID does not exist" Sep 16 18:14:07.799641 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.799621 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:14:07.804369 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:07.804347 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lf4pf"] Sep 16 18:14:08.482690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:08.482654 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" path="/var/lib/kubelet/pods/36b26290-92fd-4b0d-9d6f-d3db1a7f4daa/volumes" Sep 16 18:14:09.627612 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627571 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627875 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="storage-initializer" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627886 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="storage-initializer" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627894 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="storage-initializer" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627900 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="storage-initializer" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627928 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627934 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627944 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627950 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627957 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627962 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627974 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.627979 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.628028 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kserve-container" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.628036 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.628043 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b085d2-fca6-4284-ba23-31498e7f2b66" containerName="kube-rbac-proxy" Sep 16 18:14:09.628119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.628049 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b26290-92fd-4b0d-9d6f-d3db1a7f4daa" containerName="kserve-container" Sep 16 18:14:09.630985 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.630961 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.634184 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.634163 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\"" Sep 16 18:14:09.634686 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.634665 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:14:09.634806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.634792 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:14:09.635031 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.635009 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:14:09.635743 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.635727 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\"" Sep 16 18:14:09.637801 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.637712 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:14:09.637801 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.637744 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:14:09.659202 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.659171 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:14:09.746583 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.746544 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.746754 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.746591 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.746754 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.746736 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.746844 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.746770 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.746844 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.746799 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hscz\" (UniqueName: \"kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.847500 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.847461 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.847500 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.847505 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.847731 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.847527 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hscz\" (UniqueName: \"kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.847731 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:14:09.847618 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found Sep 16 18:14:09.847731 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.847648 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.847731 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:14:09.847675 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls podName:0978ea95-c440-47f7-941a-0a9ec42e932d nodeName:}" failed. No retries permitted until 2026-09-16 18:14:10.347660655 +0000 UTC m=+2958.290982236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-g2s5b" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d") : secret "isvc-paddle-runtime-predictor-serving-cert" not found Sep 16 18:14:09.847731 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.847708 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.848050 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.848033 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.848248 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.848230 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.848356 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.848339 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:09.860696 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:09.860665 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hscz\" (UniqueName: \"kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:10.351216 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:10.351170 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:10.353638 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:10.353614 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g2s5b\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:10.540960 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:10.540903 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:10.821200 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:10.821163 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:14:10.824626 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:14:10.824588 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0978ea95_c440_47f7_941a_0a9ec42e932d.slice/crio-2f1bf7e3da3ae6ae6e195d9dcd1283f537a9525676d5354bb53f073c0eb5d106 WatchSource:0}: Error finding container 2f1bf7e3da3ae6ae6e195d9dcd1283f537a9525676d5354bb53f073c0eb5d106: Status 404 returned error can't find the container with id 2f1bf7e3da3ae6ae6e195d9dcd1283f537a9525676d5354bb53f073c0eb5d106 Sep 16 18:14:11.790307 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:11.790272 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerStarted","Data":"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa"} Sep 16 18:14:11.790481 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:11.790330 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerStarted","Data":"2f1bf7e3da3ae6ae6e195d9dcd1283f537a9525676d5354bb53f073c0eb5d106"} Sep 16 18:14:19.827749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:19.827711 2536 generic.go:356] "Generic (PLEG): container finished" podID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerID="9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa" exitCode=0 Sep 16 18:14:19.828215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:19.827758 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerDied","Data":"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa"} Sep 16 18:14:20.834832 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:20.834801 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerStarted","Data":"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140"} Sep 16 18:14:20.835242 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:20.834864 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerStarted","Data":"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801"} Sep 16 18:14:20.835242 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:20.835076 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:20.864107 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:20.864061 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podStartSLOduration=3.606512688 podStartE2EDuration="11.864046424s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.257533736s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:14:20.863602152 +0000 UTC m=+2968.806923774" watchObservedRunningTime="2026-09-16 18:14:20.864046424 +0000 UTC m=+2968.807368027" Sep 16 18:14:21.839873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:21.839842 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:21.841398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:21.841363 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:14:22.844571 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:22.844532 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:14:27.851193 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:27.851160 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:14:27.851883 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:27.851852 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:14:37.852048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:37.852002 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:14:47.851798 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:47.851754 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:14:52.656539 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:52.656508 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:14:52.665144 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:52.665118 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:14:57.852246 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:14:57.852200 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused" Sep 16 18:15:07.853183 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:07.853151 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:15:10.769985 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:10.769952 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:15:10.770418 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:10.770212 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" containerID="cri-o://a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801" gracePeriod=30 Sep 16 18:15:10.770418 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:10.770254 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kube-rbac-proxy" containerID="cri-o://1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140" gracePeriod=30 Sep 16 18:15:11.066361 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:11.066269 2536 generic.go:356] "Generic (PLEG): container finished" podID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerID="1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140" exitCode=2 Sep 16 18:15:11.066361 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:11.066338 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerDied","Data":"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140"} Sep 16 18:15:12.845101 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:12.845049 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused" Sep 16 18:15:13.823070 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.823043 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:15:13.887377 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887285 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert\") pod \"0978ea95-c440-47f7-941a-0a9ec42e932d\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " Sep 16 18:15:13.887377 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887338 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hscz\" (UniqueName: \"kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz\") pod \"0978ea95-c440-47f7-941a-0a9ec42e932d\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " Sep 16 18:15:13.887377 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887371 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config\") pod \"0978ea95-c440-47f7-941a-0a9ec42e932d\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " Sep 16 18:15:13.887954 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887403 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location\") pod \"0978ea95-c440-47f7-941a-0a9ec42e932d\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " Sep 16 18:15:13.887954 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887420 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") pod \"0978ea95-c440-47f7-941a-0a9ec42e932d\" (UID: \"0978ea95-c440-47f7-941a-0a9ec42e932d\") " Sep 16 18:15:13.887954 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887772 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0978ea95-c440-47f7-941a-0a9ec42e932d" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:15:13.887954 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.887792 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0978ea95-c440-47f7-941a-0a9ec42e932d" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:15:13.889641 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.889615 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz" (OuterVolumeSpecName: "kube-api-access-8hscz") pod "0978ea95-c440-47f7-941a-0a9ec42e932d" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d"). InnerVolumeSpecName "kube-api-access-8hscz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:15:13.889731 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.889615 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0978ea95-c440-47f7-941a-0a9ec42e932d" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:15:13.902210 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.902170 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0978ea95-c440-47f7-941a-0a9ec42e932d" (UID: "0978ea95-c440-47f7-941a-0a9ec42e932d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:15:13.988264 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.988218 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:15:13.988264 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.988259 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hscz\" (UniqueName: \"kubernetes.io/projected/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-api-access-8hscz\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:15:13.988264 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.988272 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0978ea95-c440-47f7-941a-0a9ec42e932d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:15:13.988588 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.988285 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0978ea95-c440-47f7-941a-0a9ec42e932d-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:15:13.988588 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:13.988297 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0978ea95-c440-47f7-941a-0a9ec42e932d-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:15:14.081201 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.081164 2536 generic.go:356] "Generic (PLEG): container finished" podID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerID="a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801" exitCode=0 Sep 16 18:15:14.081398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.081218 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" Sep 16 18:15:14.081398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.081243 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerDied","Data":"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801"} Sep 16 18:15:14.081398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.081318 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b" event={"ID":"0978ea95-c440-47f7-941a-0a9ec42e932d","Type":"ContainerDied","Data":"2f1bf7e3da3ae6ae6e195d9dcd1283f537a9525676d5354bb53f073c0eb5d106"} Sep 16 18:15:14.081398 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.081356 2536 scope.go:117] "RemoveContainer" containerID="1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140" Sep 16 18:15:14.090315 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.090294 2536 scope.go:117] "RemoveContainer" containerID="a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801" Sep 16 18:15:14.098637 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.098614 2536 scope.go:117] "RemoveContainer" containerID="9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa" Sep 16 18:15:14.103282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.103259 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:15:14.106969 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.106946 2536 scope.go:117] "RemoveContainer" containerID="1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140" Sep 16 18:15:14.107667 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:15:14.107589 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140\": container with ID starting with 1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140 not found: ID does not exist" containerID="1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140" Sep 16 18:15:14.107769 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.107682 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140"} err="failed to get container status \"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140\": rpc error: code = NotFound desc = could not find container \"1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140\": container with ID starting with 1aec403fa9b8ef1532bc069f516617386f4e19450cb60ce9b84b1aff73c5e140 not found: ID does not exist" Sep 16 18:15:14.107769 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.107708 2536 scope.go:117] "RemoveContainer" containerID="a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801" Sep 16 18:15:14.108203 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:15:14.108182 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801\": container with ID starting with a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801 not found: ID does not exist" containerID="a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801" Sep 16 18:15:14.108276 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.108210 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801"} err="failed to get container status \"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801\": rpc error: code = NotFound desc = could not find container \"a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801\": container with ID starting with a740df51ee08088afccbb72a2a9512790e1ae72f60f4d9fd4031eb20dc3cc801 not found: ID does not exist" Sep 16 18:15:14.108276 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.108229 2536 scope.go:117] "RemoveContainer" containerID="9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa" Sep 16 18:15:14.108529 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:15:14.108510 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa\": container with ID starting with 9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa not found: ID does not exist" containerID="9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa" Sep 16 18:15:14.108582 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.108535 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa"} err="failed to get container status \"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa\": rpc error: code = NotFound desc = could not find container \"9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa\": container with ID starting with 9d0310a1f9b75ee93ab73563ca0c4f5d25994eeaa77cf782bfbe41fcede542fa not found: ID does not exist" Sep 16 18:15:14.109755 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.109737 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g2s5b"] Sep 16 18:15:14.478478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:14.478447 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" path="/var/lib/kubelet/pods/0978ea95-c440-47f7-941a-0a9ec42e932d/volumes" Sep 16 18:15:16.033095 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033058 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033459 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033476 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033488 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kube-rbac-proxy" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033496 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kube-rbac-proxy" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033503 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="storage-initializer" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033509 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="storage-initializer" Sep 16 18:15:16.033587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033589 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kserve-container" Sep 16 18:15:16.033977 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.033609 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="0978ea95-c440-47f7-941a-0a9ec42e932d" containerName="kube-rbac-proxy" Sep 16 18:15:16.038204 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.038183 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.041706 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.041098 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:15:16.043826 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.043746 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:15:16.044021 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.043896 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\"" Sep 16 18:15:16.044262 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.044187 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:15:16.044665 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.044645 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\"" Sep 16 18:15:16.044839 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.044819 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:15:16.044994 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.044757 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:15:16.049567 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.049542 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:15:16.108397 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.108358 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.108586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.108413 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshd9\" (UniqueName: \"kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.108586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.108445 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.108586 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.108559 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.108698 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.108594 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209441 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209407 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209441 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209442 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209686 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209481 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209686 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209529 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshd9\" (UniqueName: \"kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209686 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209570 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.209686 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:15:16.209653 2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found Sep 16 18:15:16.209900 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:15:16.209741 2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls podName:4612b4a5-b40e-4d54-804a-765a31ae70d0 nodeName:}" failed. No retries permitted until 2026-09-16 18:15:16.709719354 +0000 UTC m=+3024.653040951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found Sep 16 18:15:16.209992 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.209905 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.210152 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.210134 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.210232 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.210213 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.219465 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.219430 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshd9\" (UniqueName: \"kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.713384 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.713340 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.716002 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.715974 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:16.952109 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:16.952069 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:17.237841 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:17.237815 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:15:17.240712 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:15:17.240673 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4612b4a5_b40e_4d54_804a_765a31ae70d0.slice/crio-d7c94f6f49ae9d2b132840c227120a534598e2963bcf7334f08a73ef542a924c WatchSource:0}: Error finding container d7c94f6f49ae9d2b132840c227120a534598e2963bcf7334f08a73ef542a924c: Status 404 returned error can't find the container with id d7c94f6f49ae9d2b132840c227120a534598e2963bcf7334f08a73ef542a924c Sep 16 18:15:18.101476 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:18.101440 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerStarted","Data":"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183"} Sep 16 18:15:18.101671 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:18.101497 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerStarted","Data":"d7c94f6f49ae9d2b132840c227120a534598e2963bcf7334f08a73ef542a924c"} Sep 16 18:15:22.123374 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:22.123337 2536 generic.go:356] "Generic (PLEG): container finished" podID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerID="1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183" exitCode=0 Sep 16 18:15:22.123735 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:22.123412 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerDied","Data":"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183"} Sep 16 18:15:23.130399 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:23.130365 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerStarted","Data":"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722"} Sep 16 18:15:23.130826 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:23.130435 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerStarted","Data":"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774"} Sep 16 18:15:23.130826 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:23.130633 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:23.157021 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:23.156964 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podStartSLOduration=2.542566562 podStartE2EDuration="7.156945725s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.614379163s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:15:23.15471587 +0000 UTC m=+3031.098037518" watchObservedRunningTime="2026-09-16 18:15:23.156945725 +0000 UTC m=+3031.100267327" Sep 16 18:15:24.135268 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:24.135235 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:24.136602 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:24.136568 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:15:25.139481 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:25.139445 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:15:30.143904 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:30.143874 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:15:30.144505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:30.144484 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:15:40.145381 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:40.145340 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:15:46.992773 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:46.992693 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:15:46.996762 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:46.996740 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:46.999304 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:46.999279 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\"" Sep 16 18:15:46.999432 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:46.999324 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\"" Sep 16 18:15:47.012727 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.012704 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:15:47.069475 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.069440 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.069654 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.069490 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.069654 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.069520 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.069654 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.069546 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.069654 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.069576 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6m9\" (UniqueName: \"kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.170679 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.170637 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.170871 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.170686 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.170871 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.170717 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.170871 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.170831 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6m9\" (UniqueName: \"kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.171117 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.171098 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.171253 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.171219 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.171504 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.171485 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.171669 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.171651 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.173163 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.173147 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.180399 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.180380 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6m9\" (UniqueName: \"kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9\") pod \"isvc-pmml-runtime-predictor-885fc8777-glfvv\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.307045 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.306957 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:15:47.579749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:47.579724 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:15:47.581893 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:15:47.581860 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5f3c28_087a_418b_9c47_9ee6cd5590fe.slice/crio-85b012a3ef00dc51c971ddf1cfdf69eb9c48b9e518ddaf7757ce59da6e0792b6 WatchSource:0}: Error finding container 85b012a3ef00dc51c971ddf1cfdf69eb9c48b9e518ddaf7757ce59da6e0792b6: Status 404 returned error can't find the container with id 85b012a3ef00dc51c971ddf1cfdf69eb9c48b9e518ddaf7757ce59da6e0792b6 Sep 16 18:15:48.246597 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:48.246563 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerStarted","Data":"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72"} Sep 16 18:15:48.247064 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:48.246621 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerStarted","Data":"85b012a3ef00dc51c971ddf1cfdf69eb9c48b9e518ddaf7757ce59da6e0792b6"} Sep 16 18:15:50.144972 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:50.144905 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:15:52.273327 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:52.273293 2536 generic.go:356] "Generic (PLEG): container finished" podID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerID="c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72" exitCode=0 Sep 16 18:15:52.273695 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:15:52.273372 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerDied","Data":"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72"} Sep 16 18:16:00.145164 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:00.145123 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused" Sep 16 18:16:00.319823 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:00.319787 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerStarted","Data":"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630"} Sep 16 18:16:00.320023 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:00.319848 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerStarted","Data":"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb"} Sep 16 18:16:00.320095 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:00.320059 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:16:00.339676 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:00.339618 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podStartSLOduration=3.395248005 podStartE2EDuration="14.339602077s" totalImagesPullingTime="7.088473722s" totalInitContainerRuntime="3.85588035s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:00.338160446 +0000 UTC m=+3068.281482104" watchObservedRunningTime="2026-09-16 18:16:00.339602077 +0000 UTC m=+3068.282923710" Sep 16 18:16:01.324831 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:01.324794 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:16:01.326242 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:01.326211 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:02.329657 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:02.329615 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:07.334406 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:07.334370 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:16:07.334899 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:07.334875 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:10.145771 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:10.145741 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:16:17.314059 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:17.314020 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:16:17.314500 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:17.314362 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" containerID="cri-o://8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774" gracePeriod=30 Sep 16 18:16:17.314577 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:17.314481 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kube-rbac-proxy" containerID="cri-o://b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722" gracePeriod=30 Sep 16 18:16:17.334902 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:17.334856 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:18.403032 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:18.402997 2536 generic.go:356] "Generic (PLEG): container finished" podID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerID="b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722" exitCode=2 Sep 16 18:16:18.403418 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:18.403072 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerDied","Data":"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722"} Sep 16 18:16:20.160855 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.160828 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:16:20.266282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266186 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location\") pod \"4612b4a5-b40e-4d54-804a-765a31ae70d0\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " Sep 16 18:16:20.266282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266255 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config\") pod \"4612b4a5-b40e-4d54-804a-765a31ae70d0\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " Sep 16 18:16:20.266526 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266304 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") pod \"4612b4a5-b40e-4d54-804a-765a31ae70d0\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " Sep 16 18:16:20.266526 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266320 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert\") pod \"4612b4a5-b40e-4d54-804a-765a31ae70d0\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " Sep 16 18:16:20.266526 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266347 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bshd9\" (UniqueName: \"kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9\") pod \"4612b4a5-b40e-4d54-804a-765a31ae70d0\" (UID: \"4612b4a5-b40e-4d54-804a-765a31ae70d0\") " Sep 16 18:16:20.266698 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266676 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4612b4a5-b40e-4d54-804a-765a31ae70d0" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:16:20.266759 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.266714 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4612b4a5-b40e-4d54-804a-765a31ae70d0" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:16:20.268505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.268479 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4612b4a5-b40e-4d54-804a-765a31ae70d0" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:16:20.268505 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.268481 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9" (OuterVolumeSpecName: "kube-api-access-bshd9") pod "4612b4a5-b40e-4d54-804a-765a31ae70d0" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0"). InnerVolumeSpecName "kube-api-access-bshd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:16:20.276730 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.276700 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4612b4a5-b40e-4d54-804a-765a31ae70d0" (UID: "4612b4a5-b40e-4d54-804a-765a31ae70d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:16:20.367941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.367885 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:16:20.367941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.367934 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4612b4a5-b40e-4d54-804a-765a31ae70d0-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:16:20.367941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.367944 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4612b4a5-b40e-4d54-804a-765a31ae70d0-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:16:20.367941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.367953 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshd9\" (UniqueName: \"kubernetes.io/projected/4612b4a5-b40e-4d54-804a-765a31ae70d0-kube-api-access-bshd9\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:16:20.368212 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.367961 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4612b4a5-b40e-4d54-804a-765a31ae70d0-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:16:20.413629 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.413595 2536 generic.go:356] "Generic (PLEG): container finished" podID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerID="8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774" exitCode=0 Sep 16 18:16:20.413802 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.413652 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" Sep 16 18:16:20.413802 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.413683 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerDied","Data":"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774"} Sep 16 18:16:20.413802 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.413744 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" event={"ID":"4612b4a5-b40e-4d54-804a-765a31ae70d0","Type":"ContainerDied","Data":"d7c94f6f49ae9d2b132840c227120a534598e2963bcf7334f08a73ef542a924c"} Sep 16 18:16:20.413802 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.413773 2536 scope.go:117] "RemoveContainer" containerID="b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722" Sep 16 18:16:20.422610 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.422592 2536 scope.go:117] "RemoveContainer" containerID="8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774" Sep 16 18:16:20.430175 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.430143 2536 scope.go:117] "RemoveContainer" containerID="1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183" Sep 16 18:16:20.437042 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.437017 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:16:20.438345 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.438323 2536 scope.go:117] "RemoveContainer" containerID="b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722" Sep 16 18:16:20.438609 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:16:20.438592 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722\": container with ID starting with b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722 not found: ID does not exist" containerID="b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722" Sep 16 18:16:20.438659 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.438616 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722"} err="failed to get container status \"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722\": rpc error: code = NotFound desc = could not find container \"b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722\": container with ID starting with b09292661dd33fb0ca28eb49ccf9feeb19c5f9fa194f507a64c9af08ccbfa722 not found: ID does not exist" Sep 16 18:16:20.438659 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.438633 2536 scope.go:117] "RemoveContainer" containerID="8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774" Sep 16 18:16:20.438888 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:16:20.438873 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774\": container with ID starting with 8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774 not found: ID does not exist" containerID="8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774" Sep 16 18:16:20.438951 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.438892 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774"} err="failed to get container status \"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774\": rpc error: code = NotFound desc = could not find container \"8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774\": container with ID starting with 8094c9c68ef9ea4dfd3aca86b30acd02749a013086755c4b6d5deb0327c26774 not found: ID does not exist" Sep 16 18:16:20.438951 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.438904 2536 scope.go:117] "RemoveContainer" containerID="1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183" Sep 16 18:16:20.439151 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:16:20.439129 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183\": container with ID starting with 1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183 not found: ID does not exist" containerID="1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183" Sep 16 18:16:20.439192 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.439158 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183"} err="failed to get container status \"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183\": rpc error: code = NotFound desc = could not find container \"1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183\": container with ID starting with 1ef6090281262edfff094fbf98dc8b2a540c49d0711fd086701722034a1d8183 not found: ID does not exist" Sep 16 18:16:20.441596 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.441577 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7"] Sep 16 18:16:20.477282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:20.477254 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" path="/var/lib/kubelet/pods/4612b4a5-b40e-4d54-804a-765a31ae70d0/volumes" Sep 16 18:16:21.141366 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:21.141324 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": context deadline exceeded" Sep 16 18:16:21.145580 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:21.145552 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dshv7" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: i/o timeout" Sep 16 18:16:22.536721 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.536688 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537040 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="storage-initializer" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537052 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="storage-initializer" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537062 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kube-rbac-proxy" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537068 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kube-rbac-proxy" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537082 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537088 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537137 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kube-rbac-proxy" Sep 16 18:16:22.537142 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.537146 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4612b4a5-b40e-4d54-804a-765a31ae70d0" containerName="kserve-container" Sep 16 18:16:22.541889 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.541864 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.544556 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.544530 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\"" Sep 16 18:16:22.544688 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.544540 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\"" Sep 16 18:16:22.551623 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.551596 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:16:22.690367 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.690332 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.690554 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.690383 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.690554 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.690410 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqnl\" (UniqueName: \"kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.690554 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.690474 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.690554 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.690514 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791363 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791272 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791363 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791324 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791363 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791354 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqnl\" (UniqueName: \"kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791656 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791377 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791656 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791414 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.791835 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.791797 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.792192 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.792169 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.792284 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.792243 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.794090 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.794066 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.799868 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.799846 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqnl\" (UniqueName: \"kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl\") pod \"isvc-pmml-predictor-8674795f59-gsvsl\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:22.852900 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:22.852852 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:23.132742 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:23.130189 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:16:23.429571 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:23.429482 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerStarted","Data":"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c"} Sep 16 18:16:23.429571 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:23.429542 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerStarted","Data":"82bac177438468a24be9e464c7a0be87e8a66b7ca3c5e0ab81047937063d318d"} Sep 16 18:16:27.335567 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:27.335477 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:27.447638 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:27.447607 2536 generic.go:356] "Generic (PLEG): container finished" podID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerID="e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c" exitCode=0 Sep 16 18:16:27.447805 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:27.447683 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerDied","Data":"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c"} Sep 16 18:16:28.454911 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:28.454878 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerStarted","Data":"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861"} Sep 16 18:16:28.455309 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:28.454957 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerStarted","Data":"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587"} Sep 16 18:16:28.455309 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:28.455147 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:28.477363 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:28.477316 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podStartSLOduration=2.5313507250000002 podStartE2EDuration="6.477300206s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.945949481s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:28.475829352 +0000 UTC m=+3096.419150963" watchObservedRunningTime="2026-09-16 18:16:28.477300206 +0000 UTC m=+3096.420621809" Sep 16 18:16:29.460016 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:29.459984 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:29.461393 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:29.461369 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:16:30.465507 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:30.465466 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:16:35.470148 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:35.470114 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:16:35.470641 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:35.470615 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:16:37.335675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:37.335635 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:45.471235 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:45.471189 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:16:47.334934 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:47.334877 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:16:55.470820 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:55.470780 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:16:57.335382 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:16:57.335339 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:17:05.470754 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:05.470715 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:17:07.335105 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:07.335066 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:17:15.471304 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:15.471213 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:17:15.472796 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:15.472755 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused" Sep 16 18:17:25.471155 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:25.471117 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:17:25.473975 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:25.473952 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:17:28.122457 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:28.122426 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:17:28.122933 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:28.122708 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" containerID="cri-o://a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb" gracePeriod=30 Sep 16 18:17:28.122933 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:28.122728 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kube-rbac-proxy" containerID="cri-o://22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630" gracePeriod=30 Sep 16 18:17:28.725352 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:28.725321 2536 generic.go:356] "Generic (PLEG): container finished" podID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerID="22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630" exitCode=2 Sep 16 18:17:28.725534 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:28.725359 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerDied","Data":"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630"} Sep 16 18:17:31.965072 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:31.965046 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:17:32.006119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006083 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls\") pod \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " Sep 16 18:17:32.006282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006132 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs6m9\" (UniqueName: \"kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9\") pod \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " Sep 16 18:17:32.006282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006189 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert\") pod \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " Sep 16 18:17:32.006282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006219 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config\") pod \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " Sep 16 18:17:32.006282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006237 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location\") pod \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\" (UID: \"cb5f3c28-087a-418b-9c47-9ee6cd5590fe\") " Sep 16 18:17:32.006584 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006562 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb5f3c28-087a-418b-9c47-9ee6cd5590fe" (UID: "cb5f3c28-087a-418b-9c47-9ee6cd5590fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:17:32.006652 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006598 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb5f3c28-087a-418b-9c47-9ee6cd5590fe" (UID: "cb5f3c28-087a-418b-9c47-9ee6cd5590fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:17:32.006652 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.006593 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb5f3c28-087a-418b-9c47-9ee6cd5590fe" (UID: "cb5f3c28-087a-418b-9c47-9ee6cd5590fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:17:32.008263 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.008235 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9" (OuterVolumeSpecName: "kube-api-access-cs6m9") pod "cb5f3c28-087a-418b-9c47-9ee6cd5590fe" (UID: "cb5f3c28-087a-418b-9c47-9ee6cd5590fe"). InnerVolumeSpecName "kube-api-access-cs6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:17:32.008370 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.008297 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb5f3c28-087a-418b-9c47-9ee6cd5590fe" (UID: "cb5f3c28-087a-418b-9c47-9ee6cd5590fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:17:32.106974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.106867 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:32.106974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.106898 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:32.106974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.106907 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:32.106974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.106938 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:32.106974 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.106947 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs6m9\" (UniqueName: \"kubernetes.io/projected/cb5f3c28-087a-418b-9c47-9ee6cd5590fe-kube-api-access-cs6m9\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:32.745281 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.745194 2536 generic.go:356] "Generic (PLEG): container finished" podID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerID="a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb" exitCode=0 Sep 16 18:17:32.745281 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.745248 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" Sep 16 18:17:32.745462 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.745251 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerDied","Data":"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb"} Sep 16 18:17:32.745462 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.745377 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv" event={"ID":"cb5f3c28-087a-418b-9c47-9ee6cd5590fe","Type":"ContainerDied","Data":"85b012a3ef00dc51c971ddf1cfdf69eb9c48b9e518ddaf7757ce59da6e0792b6"} Sep 16 18:17:32.745462 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.745408 2536 scope.go:117] "RemoveContainer" containerID="22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630" Sep 16 18:17:32.753567 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.753549 2536 scope.go:117] "RemoveContainer" containerID="a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb" Sep 16 18:17:32.763207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.762896 2536 scope.go:117] "RemoveContainer" containerID="c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72" Sep 16 18:17:32.767561 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.767537 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:17:32.771022 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771003 2536 scope.go:117] "RemoveContainer" containerID="22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630" Sep 16 18:17:32.771321 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:32.771302 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630\": container with ID starting with 22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630 not found: ID does not exist" containerID="22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630" Sep 16 18:17:32.771411 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771332 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630"} err="failed to get container status \"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630\": rpc error: code = NotFound desc = could not find container \"22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630\": container with ID starting with 22f6b7b1a4265c1cc2658a49628500e7ab7f6d18f0baa27bacc0901397d22630 not found: ID does not exist" Sep 16 18:17:32.771411 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771356 2536 scope.go:117] "RemoveContainer" containerID="a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb" Sep 16 18:17:32.771506 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771421 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-glfvv"] Sep 16 18:17:32.771653 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:32.771636 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb\": container with ID starting with a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb not found: ID does not exist" containerID="a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb" Sep 16 18:17:32.771689 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771660 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb"} err="failed to get container status \"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb\": rpc error: code = NotFound desc = could not find container \"a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb\": container with ID starting with a1d76b3ff0a13e396d642811e8543ebf430bfe71e1a400186cd248a619c390fb not found: ID does not exist" Sep 16 18:17:32.771689 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771677 2536 scope.go:117] "RemoveContainer" containerID="c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72" Sep 16 18:17:32.771875 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:32.771861 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72\": container with ID starting with c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72 not found: ID does not exist" containerID="c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72" Sep 16 18:17:32.771911 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:32.771878 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72"} err="failed to get container status \"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72\": rpc error: code = NotFound desc = could not find container \"c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72\": container with ID starting with c2677c3dcad33111cd12fae06dbe87b9af3c360e91d2fa82d5c34b8858973c72 not found: ID does not exist" Sep 16 18:17:33.322105 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322073 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322408 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322418 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322427 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kube-rbac-proxy" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322432 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kube-rbac-proxy" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322443 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="storage-initializer" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322449 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="storage-initializer" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322504 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kserve-container" Sep 16 18:17:33.322517 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.322511 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" containerName="kube-rbac-proxy" Sep 16 18:17:33.325322 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.325304 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.327951 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.327911 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\"" Sep 16 18:17:33.328082 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.327965 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\"" Sep 16 18:17:33.335943 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.335891 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:17:33.418277 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.418238 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.418277 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.418278 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.418509 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.418320 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.418509 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.418388 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.418509 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.418446 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tt6\" (UniqueName: \"kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.519882 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.519846 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.519882 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.519880 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520135 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.519905 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520135 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.519969 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520135 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.520019 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tt6\" (UniqueName: \"kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520446 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.520419 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.520726 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.520800 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.520726 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.522342 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.522314 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.529463 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.529441 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tt6\" (UniqueName: \"kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.636139 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.636054 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:33.906557 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:33.906529 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:17:33.908350 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:17:33.908328 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702dd6bc_91c5_49d6_a895_c05455c13f95.slice/crio-f29e0644ec9b71e5fd6d9d189af73eb5e771b6a824d59d605bc9d408ef44a3cf WatchSource:0}: Error finding container f29e0644ec9b71e5fd6d9d189af73eb5e771b6a824d59d605bc9d408ef44a3cf: Status 404 returned error can't find the container with id f29e0644ec9b71e5fd6d9d189af73eb5e771b6a824d59d605bc9d408ef44a3cf Sep 16 18:17:34.478210 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:34.478168 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5f3c28-087a-418b-9c47-9ee6cd5590fe" path="/var/lib/kubelet/pods/cb5f3c28-087a-418b-9c47-9ee6cd5590fe/volumes" Sep 16 18:17:34.756256 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:34.756158 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerStarted","Data":"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b"} Sep 16 18:17:34.756256 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:34.756228 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerStarted","Data":"f29e0644ec9b71e5fd6d9d189af73eb5e771b6a824d59d605bc9d408ef44a3cf"} Sep 16 18:17:35.470678 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:35.470637 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:17:38.777829 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:38.777795 2536 generic.go:356] "Generic (PLEG): container finished" podID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerID="ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b" exitCode=0 Sep 16 18:17:38.778297 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:38.777876 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerDied","Data":"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b"} Sep 16 18:17:39.784706 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.784669 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerStarted","Data":"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083"} Sep 16 18:17:39.785126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.784732 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerStarted","Data":"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478"} Sep 16 18:17:39.785126 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.785002 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:39.785236 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.785124 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:39.786431 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.786405 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:17:39.806796 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:39.806744 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podStartSLOduration=2.8427438179999998 podStartE2EDuration="6.8067289s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.963985082s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:39.805683175 +0000 UTC m=+3167.749004801" watchObservedRunningTime="2026-09-16 18:17:39.8067289 +0000 UTC m=+3167.750050500" Sep 16 18:17:40.790237 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:40.790190 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:17:45.471222 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:45.471181 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused" Sep 16 18:17:45.795281 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:45.795198 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:17:45.795733 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:45.795701 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:17:50.477222 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:50.477196 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:17:53.823307 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:53.823271 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:17:53.823722 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:53.823505 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" containerID="cri-o://d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587" gracePeriod=30 Sep 16 18:17:53.823722 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:53.823552 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kube-rbac-proxy" containerID="cri-o://4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861" gracePeriod=30 Sep 16 18:17:54.853033 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:54.853004 2536 generic.go:356] "Generic (PLEG): container finished" podID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerID="4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861" exitCode=2 Sep 16 18:17:54.853415 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:54.853085 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerDied","Data":"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861"} Sep 16 18:17:55.466475 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:55.466429 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused" Sep 16 18:17:55.795797 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:55.795746 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:17:57.570800 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.570768 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:17:57.635675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.635642 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls\") pod \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " Sep 16 18:17:57.635675 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.635677 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dqnl\" (UniqueName: \"kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl\") pod \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " Sep 16 18:17:57.635886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.635808 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert\") pod \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " Sep 16 18:17:57.635886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.635838 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config\") pod \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " Sep 16 18:17:57.635886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.635868 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location\") pod \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\" (UID: \"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c\") " Sep 16 18:17:57.636204 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.636176 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" (UID: "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:17:57.636204 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.636193 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" (UID: "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:17:57.636298 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.636209 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" (UID: "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:17:57.637806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.637786 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl" (OuterVolumeSpecName: "kube-api-access-2dqnl") pod "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" (UID: "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c"). InnerVolumeSpecName "kube-api-access-2dqnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:17:57.637899 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.637879 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" (UID: "ab9db19c-5985-4fb9-a06d-5b87a8b2a68c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:17:57.737028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.736990 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:57.737028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.737022 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dqnl\" (UniqueName: \"kubernetes.io/projected/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-api-access-2dqnl\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:57.737028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.737032 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:57.737256 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.737041 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:57.737256 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.737050 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:17:57.867857 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.867823 2536 generic.go:356] "Generic (PLEG): container finished" podID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerID="d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587" exitCode=0 Sep 16 18:17:57.868068 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.867880 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" Sep 16 18:17:57.868068 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.867898 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerDied","Data":"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587"} Sep 16 18:17:57.868068 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.867973 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl" event={"ID":"ab9db19c-5985-4fb9-a06d-5b87a8b2a68c","Type":"ContainerDied","Data":"82bac177438468a24be9e464c7a0be87e8a66b7ca3c5e0ab81047937063d318d"} Sep 16 18:17:57.868068 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.868001 2536 scope.go:117] "RemoveContainer" containerID="4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861" Sep 16 18:17:57.876663 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.876647 2536 scope.go:117] "RemoveContainer" containerID="d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587" Sep 16 18:17:57.884368 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.884353 2536 scope.go:117] "RemoveContainer" containerID="e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c" Sep 16 18:17:57.892259 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892218 2536 scope.go:117] "RemoveContainer" containerID="4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861" Sep 16 18:17:57.892402 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892379 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:17:57.892534 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:57.892515 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861\": container with ID starting with 4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861 not found: ID does not exist" containerID="4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861" Sep 16 18:17:57.892589 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892543 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861"} err="failed to get container status \"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861\": rpc error: code = NotFound desc = could not find container \"4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861\": container with ID starting with 4a48962456c3d809c96a943af7c8cf6fd032bc49c1509c1cd8daf344f3e20861 not found: ID does not exist" Sep 16 18:17:57.892589 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892562 2536 scope.go:117] "RemoveContainer" containerID="d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587" Sep 16 18:17:57.892834 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:57.892817 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587\": container with ID starting with d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587 not found: ID does not exist" containerID="d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587" Sep 16 18:17:57.892877 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892841 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587"} err="failed to get container status \"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587\": rpc error: code = NotFound desc = could not find container \"d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587\": container with ID starting with d8792e549cd038eee6ec530c9b05a8a86dadbbb68dfdfc047495a111dea49587 not found: ID does not exist" Sep 16 18:17:57.892877 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.892856 2536 scope.go:117] "RemoveContainer" containerID="e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c" Sep 16 18:17:57.893079 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:17:57.893061 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c\": container with ID starting with e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c not found: ID does not exist" containerID="e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c" Sep 16 18:17:57.893119 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.893082 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c"} err="failed to get container status \"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c\": rpc error: code = NotFound desc = could not find container \"e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c\": container with ID starting with e7bfb47fbe09929c3ea76f10d61529de8e73cb26f6d1cb293ad321d5c41b298c not found: ID does not exist" Sep 16 18:17:57.899059 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:57.899039 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-gsvsl"] Sep 16 18:17:58.477791 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:17:58.477760 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" path="/var/lib/kubelet/pods/ab9db19c-5985-4fb9-a06d-5b87a8b2a68c/volumes" Sep 16 18:18:05.796171 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:05.796125 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:18:15.796201 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:15.796159 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:18:25.796013 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:25.795971 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:18:35.796251 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:35.796164 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:18:45.795990 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:45.795877 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:18:55.796166 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:18:55.796128 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:19:05.796065 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:05.796037 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:19:14.332765 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:14.332733 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:19:14.333259 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:14.332992 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" containerID="cri-o://7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478" gracePeriod=30 Sep 16 18:19:14.333259 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:14.333060 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kube-rbac-proxy" containerID="cri-o://01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083" gracePeriod=30 Sep 16 18:19:15.202870 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:15.202837 2536 generic.go:356] "Generic (PLEG): container finished" podID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerID="01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083" exitCode=2 Sep 16 18:19:15.203048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:15.202928 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerDied","Data":"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083"} Sep 16 18:19:15.791502 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:15.791455 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused" Sep 16 18:19:15.795951 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:15.795904 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused" Sep 16 18:19:18.073373 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.073351 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:19:18.213269 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213185 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location\") pod \"702dd6bc-91c5-49d6-a895-c05455c13f95\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " Sep 16 18:19:18.213269 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213238 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert\") pod \"702dd6bc-91c5-49d6-a895-c05455c13f95\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " Sep 16 18:19:18.213478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213297 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config\") pod \"702dd6bc-91c5-49d6-a895-c05455c13f95\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " Sep 16 18:19:18.213478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213315 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8tt6\" (UniqueName: \"kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6\") pod \"702dd6bc-91c5-49d6-a895-c05455c13f95\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " Sep 16 18:19:18.213478 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213348 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls\") pod \"702dd6bc-91c5-49d6-a895-c05455c13f95\" (UID: \"702dd6bc-91c5-49d6-a895-c05455c13f95\") " Sep 16 18:19:18.213640 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213567 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "702dd6bc-91c5-49d6-a895-c05455c13f95" (UID: "702dd6bc-91c5-49d6-a895-c05455c13f95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:19:18.213705 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213637 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "702dd6bc-91c5-49d6-a895-c05455c13f95" (UID: "702dd6bc-91c5-49d6-a895-c05455c13f95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:19:18.213705 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.213663 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "702dd6bc-91c5-49d6-a895-c05455c13f95" (UID: "702dd6bc-91c5-49d6-a895-c05455c13f95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:19:18.215568 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.215539 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6" (OuterVolumeSpecName: "kube-api-access-b8tt6") pod "702dd6bc-91c5-49d6-a895-c05455c13f95" (UID: "702dd6bc-91c5-49d6-a895-c05455c13f95"). InnerVolumeSpecName "kube-api-access-b8tt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:19:18.215706 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.215624 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "702dd6bc-91c5-49d6-a895-c05455c13f95" (UID: "702dd6bc-91c5-49d6-a895-c05455c13f95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:19:18.218202 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.218178 2536 generic.go:356] "Generic (PLEG): container finished" podID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerID="7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478" exitCode=0 Sep 16 18:19:18.218333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.218224 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerDied","Data":"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478"} Sep 16 18:19:18.218333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.218260 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" Sep 16 18:19:18.218333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.218288 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh" event={"ID":"702dd6bc-91c5-49d6-a895-c05455c13f95","Type":"ContainerDied","Data":"f29e0644ec9b71e5fd6d9d189af73eb5e771b6a824d59d605bc9d408ef44a3cf"} Sep 16 18:19:18.218333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.218324 2536 scope.go:117] "RemoveContainer" containerID="01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083" Sep 16 18:19:18.233609 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.233580 2536 scope.go:117] "RemoveContainer" containerID="7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478" Sep 16 18:19:18.241318 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.241298 2536 scope.go:117] "RemoveContainer" containerID="ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b" Sep 16 18:19:18.246710 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.246687 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:19:18.249296 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.249275 2536 scope.go:117] "RemoveContainer" containerID="01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083" Sep 16 18:19:18.249784 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:19:18.249761 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083\": container with ID starting with 01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083 not found: ID does not exist" containerID="01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083" Sep 16 18:19:18.249886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.249796 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083"} err="failed to get container status \"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083\": rpc error: code = NotFound desc = could not find container \"01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083\": container with ID starting with 01d68c397859fe959c0e1f2ae95985edbb80e034f6740da6347516e99310c083 not found: ID does not exist" Sep 16 18:19:18.249886 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.249820 2536 scope.go:117] "RemoveContainer" containerID="7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478" Sep 16 18:19:18.250171 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:19:18.250148 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478\": container with ID starting with 7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478 not found: ID does not exist" containerID="7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478" Sep 16 18:19:18.250215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.250183 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478"} err="failed to get container status \"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478\": rpc error: code = NotFound desc = could not find container \"7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478\": container with ID starting with 7bf8f6da6c70022bc0f8f38c4ea3219d44540ee4f2ccaf9b15926d1893770478 not found: ID does not exist" Sep 16 18:19:18.250215 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.250202 2536 scope.go:117] "RemoveContainer" containerID="ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b" Sep 16 18:19:18.250499 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:19:18.250479 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b\": container with ID starting with ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b not found: ID does not exist" containerID="ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b" Sep 16 18:19:18.250587 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.250503 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b"} err="failed to get container status \"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b\": rpc error: code = NotFound desc = could not find container \"ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b\": container with ID starting with ee1dbc1a450d06dd3af6c5ef9ce4cf2fe2b50f29f8228d031cfb590d4f02e64b not found: ID does not exist" Sep 16 18:19:18.251278 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.251260 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-cmgxh"] Sep 16 18:19:18.314631 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.314599 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:19:18.314631 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.314626 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tt6\" (UniqueName: \"kubernetes.io/projected/702dd6bc-91c5-49d6-a895-c05455c13f95-kube-api-access-b8tt6\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:19:18.314631 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.314636 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/702dd6bc-91c5-49d6-a895-c05455c13f95-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:19:18.314849 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.314645 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/702dd6bc-91c5-49d6-a895-c05455c13f95-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:19:18.314849 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.314653 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/702dd6bc-91c5-49d6-a895-c05455c13f95-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:19:18.478043 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:18.478008 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" path="/var/lib/kubelet/pods/702dd6bc-91c5-49d6-a895-c05455c13f95/volumes" Sep 16 18:19:19.536116 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536076 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536406 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536419 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536425 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kube-rbac-proxy" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536431 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kube-rbac-proxy" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536439 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="storage-initializer" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536444 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="storage-initializer" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536453 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536457 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" Sep 16 18:19:19.536461 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536465 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kube-rbac-proxy" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536471 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kube-rbac-proxy" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536477 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="storage-initializer" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536482 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="storage-initializer" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536542 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kserve-container" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536550 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kserve-container" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536556 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9db19c-5985-4fb9-a06d-5b87a8b2a68c" containerName="kube-rbac-proxy" Sep 16 18:19:19.536747 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.536561 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="702dd6bc-91c5-49d6-a895-c05455c13f95" containerName="kube-rbac-proxy" Sep 16 18:19:19.539569 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.539542 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.542529 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.542506 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:19:19.543504 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543484 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\"" Sep 16 18:19:19.543504 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543501 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2f74f8-predictor-serving-cert\"" Sep 16 18:19:19.543848 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543828 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:19:19.543941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543833 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:19:19.543941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543880 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2f74f8-kube-rbac-proxy-sar-config\"" Sep 16 18:19:19.543941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.543893 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:19:19.550479 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.550458 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:19:19.626498 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.626466 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.626713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.626513 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.626713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.626598 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrn2\" (UniqueName: \"kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.626713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.626633 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.626713 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.626653 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727050 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727015 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727223 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727070 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrn2\" (UniqueName: \"kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727223 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727095 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727223 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727112 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727223 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727149 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.727650 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.727625 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.728048 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.728027 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.728106 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.728027 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.729413 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.729387 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.737154 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.737136 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrn2\" (UniqueName: \"kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2\") pod \"isvc-primary-2f74f8-predictor-865fd84c87-2rvw9\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:19.851143 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:19.851056 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:20.122941 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:20.122899 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:19:20.124635 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:19:20.124609 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e29c91a_a2a0_4a3e_8c38_8347ae32e412.slice/crio-da1925d6f1db2a423b87ceb4a19a19609f3e0003f5eec7494b838b9c1728efe6 WatchSource:0}: Error finding container da1925d6f1db2a423b87ceb4a19a19609f3e0003f5eec7494b838b9c1728efe6: Status 404 returned error can't find the container with id da1925d6f1db2a423b87ceb4a19a19609f3e0003f5eec7494b838b9c1728efe6 Sep 16 18:19:20.230071 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:20.230031 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerStarted","Data":"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c"} Sep 16 18:19:20.230267 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:20.230107 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerStarted","Data":"da1925d6f1db2a423b87ceb4a19a19609f3e0003f5eec7494b838b9c1728efe6"} Sep 16 18:19:25.252555 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:25.252519 2536 generic.go:356] "Generic (PLEG): container finished" podID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerID="d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c" exitCode=0 Sep 16 18:19:25.253017 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:25.252592 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerDied","Data":"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c"} Sep 16 18:19:26.264184 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.264152 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerStarted","Data":"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf"} Sep 16 18:19:26.264601 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.264232 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerStarted","Data":"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47"} Sep 16 18:19:26.264601 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.264485 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:26.264695 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.264622 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:26.266055 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.266032 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:19:26.288049 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:26.288002 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podStartSLOduration=3.027809664 podStartE2EDuration="7.287988571s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.260178907s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:19:26.286356053 +0000 UTC m=+3274.229677672" watchObservedRunningTime="2026-09-16 18:19:26.287988571 +0000 UTC m=+3274.231310172" Sep 16 18:19:27.268865 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:27.268821 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:19:32.273659 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:32.273630 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:19:32.274112 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:32.274088 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:19:42.274549 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:42.274509 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:19:52.275101 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:52.275059 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:19:52.683111 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:52.683026 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:19:52.699418 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:19:52.699391 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:20:02.275073 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:20:02.275028 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:20:12.274171 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:20:12.274134 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:20:22.275121 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:20:22.275037 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:20:32.274703 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:20:32.274663 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused" Sep 16 18:20:42.275091 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:20:42.275056 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:21:04.763977 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.763941 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:04.767164 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.767149 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.769911 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.769889 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2f74f8-predictor-serving-cert\"" Sep 16 18:21:04.770428 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.770411 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-2f74f8-dockercfg-jcndg\"" Sep 16 18:21:04.770428 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.770422 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2f74f8-kube-rbac-proxy-sar-config\"" Sep 16 18:21:04.770685 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.770669 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-2f74f8\"" Sep 16 18:21:04.781804 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.781776 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:04.898247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.898213 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.898247 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.898249 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.898442 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.898323 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.898442 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.898363 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5tp\" (UniqueName: \"kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.898442 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.898387 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999477 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999448 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999477 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999482 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999703 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999523 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999703 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999554 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5tp\" (UniqueName: \"kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999703 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999577 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:04.999967 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:04.999945 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.000348 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.000318 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.000460 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.000385 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.002094 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.002071 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.009804 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.009778 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5tp\" (UniqueName: \"kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp\") pod \"isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.076786 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.076692 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:05.348391 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.348364 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:05.350689 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:21:05.350664 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f568a1_0c02_435a_b068_6dbb8a616692.slice/crio-13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901 WatchSource:0}: Error finding container 13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901: Status 404 returned error can't find the container with id 13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901 Sep 16 18:21:05.689204 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.689108 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerStarted","Data":"71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745"} Sep 16 18:21:05.689204 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:05.689175 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerStarted","Data":"13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901"} Sep 16 18:21:11.718581 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:11.718555 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/0.log" Sep 16 18:21:11.719047 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:11.718593 2536 generic.go:356] "Generic (PLEG): container finished" podID="94f568a1-0c02-435a-b068-6dbb8a616692" containerID="71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745" exitCode=1 Sep 16 18:21:11.719047 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:11.718650 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerDied","Data":"71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745"} Sep 16 18:21:12.725298 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:12.725269 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/0.log" Sep 16 18:21:12.725773 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:12.725351 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerStarted","Data":"f2eb030625ab6713a3b89af01d0506b90f13042b6ea5891fa1536b5df9757467"} Sep 16 18:21:14.735225 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.735201 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/1.log" Sep 16 18:21:14.735564 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.735550 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/0.log" Sep 16 18:21:14.735611 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.735583 2536 generic.go:356] "Generic (PLEG): container finished" podID="94f568a1-0c02-435a-b068-6dbb8a616692" containerID="f2eb030625ab6713a3b89af01d0506b90f13042b6ea5891fa1536b5df9757467" exitCode=1 Sep 16 18:21:14.735646 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.735630 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerDied","Data":"f2eb030625ab6713a3b89af01d0506b90f13042b6ea5891fa1536b5df9757467"} Sep 16 18:21:14.735690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.735680 2536 scope.go:117] "RemoveContainer" containerID="71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745" Sep 16 18:21:14.736072 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.736052 2536 scope.go:117] "RemoveContainer" containerID="71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745" Sep 16 18:21:14.767163 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:21:14.767130 2536 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_kserve-ci-e2e-test(94f568a1-0c02-435a-b068-6dbb8a616692)\"" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" Sep 16 18:21:14.767281 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:21:14.767241 2536 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_kserve-ci-e2e-test_94f568a1-0c02-435a-b068-6dbb8a616692_0 in pod sandbox 13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901 from index: no such id: '71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745'" containerID="71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745" Sep 16 18:21:14.767328 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:14.767282 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_kserve-ci-e2e-test_94f568a1-0c02-435a-b068-6dbb8a616692_0 in pod sandbox 13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901 from index: no such id: '71daeaace0443cb89fe57d7691f5d6f0172dcbba7d62292184865f97714c3745'" Sep 16 18:21:15.740962 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:15.740897 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/1.log" Sep 16 18:21:22.767099 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.767067 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:22.803967 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.803909 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:21:22.804286 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.804254 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" containerID="cri-o://cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47" gracePeriod=30 Sep 16 18:21:22.804576 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.804283 2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kube-rbac-proxy" containerID="cri-o://b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf" gracePeriod=30 Sep 16 18:21:22.924880 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.924859 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/1.log" Sep 16 18:21:22.927011 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.926996 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:22.951849 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.951819 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location\") pod \"94f568a1-0c02-435a-b068-6dbb8a616692\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " Sep 16 18:21:22.952028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.951881 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq5tp\" (UniqueName: \"kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp\") pod \"94f568a1-0c02-435a-b068-6dbb8a616692\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " Sep 16 18:21:22.952028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.951963 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls\") pod \"94f568a1-0c02-435a-b068-6dbb8a616692\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " Sep 16 18:21:22.952028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.951990 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config\") pod \"94f568a1-0c02-435a-b068-6dbb8a616692\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " Sep 16 18:21:22.952028 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.952014 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert\") pod \"94f568a1-0c02-435a-b068-6dbb8a616692\" (UID: \"94f568a1-0c02-435a-b068-6dbb8a616692\") " Sep 16 18:21:22.952246 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.952107 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94f568a1-0c02-435a-b068-6dbb8a616692" (UID: "94f568a1-0c02-435a-b068-6dbb8a616692"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:21:22.952309 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.952270 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94f568a1-0c02-435a-b068-6dbb8a616692-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:22.952404 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.952379 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94f568a1-0c02-435a-b068-6dbb8a616692" (UID: "94f568a1-0c02-435a-b068-6dbb8a616692"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:21:22.952480 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.952459 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94f568a1-0c02-435a-b068-6dbb8a616692" (UID: "94f568a1-0c02-435a-b068-6dbb8a616692"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:21:22.953981 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.953960 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94f568a1-0c02-435a-b068-6dbb8a616692" (UID: "94f568a1-0c02-435a-b068-6dbb8a616692"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:21:22.954077 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:22.954059 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp" (OuterVolumeSpecName: "kube-api-access-qq5tp") pod "94f568a1-0c02-435a-b068-6dbb8a616692" (UID: "94f568a1-0c02-435a-b068-6dbb8a616692"). InnerVolumeSpecName "kube-api-access-qq5tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:21:23.053380 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.053290 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94f568a1-0c02-435a-b068-6dbb8a616692-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:23.053380 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.053319 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:23.053380 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.053334 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94f568a1-0c02-435a-b068-6dbb8a616692-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:23.053380 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.053345 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5tp\" (UniqueName: \"kubernetes.io/projected/94f568a1-0c02-435a-b068-6dbb8a616692-kube-api-access-qq5tp\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:23.776749 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.776715 2536 generic.go:356] "Generic (PLEG): container finished" podID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerID="b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf" exitCode=2 Sep 16 18:21:23.777207 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.776788 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerDied","Data":"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf"} Sep 16 18:21:23.778039 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.778022 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l_94f568a1-0c02-435a-b068-6dbb8a616692/storage-initializer/1.log" Sep 16 18:21:23.778139 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.778075 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" event={"ID":"94f568a1-0c02-435a-b068-6dbb8a616692","Type":"ContainerDied","Data":"13e2778b0019d627747f40a155cc21bddbafc03a7cae8c98dc6e4a4f99f19901"} Sep 16 18:21:23.778139 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.778102 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l" Sep 16 18:21:23.778139 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.778119 2536 scope.go:117] "RemoveContainer" containerID="f2eb030625ab6713a3b89af01d0506b90f13042b6ea5891fa1536b5df9757467" Sep 16 18:21:23.832930 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.832888 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:23.837391 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:23.837364 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2f74f8-predictor-65d9c6df8d-5sp9l"] Sep 16 18:21:24.477564 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:24.477533 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" path="/var/lib/kubelet/pods/94f568a1-0c02-435a-b068-6dbb8a616692/volumes" Sep 16 18:21:27.269412 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.269368 2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused" Sep 16 18:21:27.549690 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.549665 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:21:27.596673 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.596646 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config\") pod \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " Sep 16 18:21:27.596845 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.596687 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls\") pod \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " Sep 16 18:21:27.596845 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.596709 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location\") pod \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " Sep 16 18:21:27.596845 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.596735 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xrn2\" (UniqueName: \"kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2\") pod \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " Sep 16 18:21:27.596845 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.596781 2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert\") pod \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\" (UID: \"4e29c91a-a2a0-4a3e-8c38-8347ae32e412\") " Sep 16 18:21:27.597096 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.597067 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e29c91a-a2a0-4a3e-8c38-8347ae32e412" (UID: "4e29c91a-a2a0-4a3e-8c38-8347ae32e412"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:21:27.597253 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.597231 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e29c91a-a2a0-4a3e-8c38-8347ae32e412" (UID: "4e29c91a-a2a0-4a3e-8c38-8347ae32e412"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:21:27.597318 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.597275 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e29c91a-a2a0-4a3e-8c38-8347ae32e412" (UID: "4e29c91a-a2a0-4a3e-8c38-8347ae32e412"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:21:27.598944 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.598908 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2" (OuterVolumeSpecName: "kube-api-access-6xrn2") pod "4e29c91a-a2a0-4a3e-8c38-8347ae32e412" (UID: "4e29c91a-a2a0-4a3e-8c38-8347ae32e412"). InnerVolumeSpecName "kube-api-access-6xrn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:21:27.598944 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.598910 2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e29c91a-a2a0-4a3e-8c38-8347ae32e412" (UID: "4e29c91a-a2a0-4a3e-8c38-8347ae32e412"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:21:27.697627 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.697588 2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-cabundle-cert\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:27.697627 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.697621 2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:27.697627 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.697631 2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-proxy-tls\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:27.697858 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.697639 2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kserve-provision-location\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:27.697858 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.697649 2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrn2\" (UniqueName: \"kubernetes.io/projected/4e29c91a-a2a0-4a3e-8c38-8347ae32e412-kube-api-access-6xrn2\") on node \"ip-10-0-130-222.ec2.internal\" DevicePath \"\"" Sep 16 18:21:27.798467 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.798370 2536 generic.go:356] "Generic (PLEG): container finished" podID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerID="cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47" exitCode=0 Sep 16 18:21:27.798467 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.798430 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerDied","Data":"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47"} Sep 16 18:21:27.798467 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.798434 2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" Sep 16 18:21:27.798745 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.798477 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9" event={"ID":"4e29c91a-a2a0-4a3e-8c38-8347ae32e412","Type":"ContainerDied","Data":"da1925d6f1db2a423b87ceb4a19a19609f3e0003f5eec7494b838b9c1728efe6"} Sep 16 18:21:27.798745 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.798505 2536 scope.go:117] "RemoveContainer" containerID="b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf" Sep 16 18:21:27.807419 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.807401 2536 scope.go:117] "RemoveContainer" containerID="cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47" Sep 16 18:21:27.814692 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.814677 2536 scope.go:117] "RemoveContainer" containerID="d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c" Sep 16 18:21:27.821620 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.821599 2536 scope.go:117] "RemoveContainer" containerID="b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf" Sep 16 18:21:27.821891 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:21:27.821873 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf\": container with ID starting with b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf not found: ID does not exist" containerID="b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf" Sep 16 18:21:27.821982 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.821903 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf"} err="failed to get container status \"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf\": rpc error: code = NotFound desc = could not find container \"b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf\": container with ID starting with b3bf7240a0d1f891223303429e128ebb8043b60c675f2d56164b02cfef9b51cf not found: ID does not exist" Sep 16 18:21:27.821982 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.821941 2536 scope.go:117] "RemoveContainer" containerID="cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47" Sep 16 18:21:27.822207 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:21:27.822189 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47\": container with ID starting with cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47 not found: ID does not exist" containerID="cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47" Sep 16 18:21:27.822246 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.822213 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47"} err="failed to get container status \"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47\": rpc error: code = NotFound desc = could not find container \"cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47\": container with ID starting with cfe11fc9d1b5b51f61f944e9863848b1ea70853dbc3b8e20276070044f09af47 not found: ID does not exist" Sep 16 18:21:27.822246 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.822229 2536 scope.go:117] "RemoveContainer" containerID="d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c" Sep 16 18:21:27.822459 ip-10-0-130-222 kubenswrapper[2536]: E0916 18:21:27.822442 2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c\": container with ID starting with d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c not found: ID does not exist" containerID="d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c" Sep 16 18:21:27.822518 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.822468 2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c"} err="failed to get container status \"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c\": rpc error: code = NotFound desc = could not find container \"d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c\": container with ID starting with d1d1e34e702aead6c71e71d420d21bd2394c0710cdefeb4d24d4f281ad56f04c not found: ID does not exist" Sep 16 18:21:27.830611 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.830586 2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:21:27.837498 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:27.837476 2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2f74f8-predictor-865fd84c87-2rvw9"] Sep 16 18:21:28.477468 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:21:28.477428 2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" path="/var/lib/kubelet/pods/4e29c91a-a2a0-4a3e-8c38-8347ae32e412/volumes" Sep 16 18:22:05.327928 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:05.327896 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t677q_a839c16c-c00b-428c-8c7e-1b25b469967f/global-pull-secret-syncer/0.log" Sep 16 18:22:05.456289 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:05.456261 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p4kfb_e3a95030-580d-4d9f-b746-3359da873540/konnectivity-agent/0.log" Sep 16 18:22:05.500953 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:05.500927 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-222.ec2.internal_3cab83a51988543fe63346f62bfb0589/haproxy/0.log" Sep 16 18:22:10.044377 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:10.044348 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6858ddbdb-cxvw7_61a5a0dd-93a3-42b8-8332-4798a0728698/metrics-server/0.log" Sep 16 18:22:10.091696 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:10.091673 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-mzs92_90aae6d6-be66-435d-a79a-e2115d70ca1b/monitoring-plugin/0.log" Sep 16 18:22:10.354871 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:10.354792 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzk4k_e6e500d3-c03a-4a33-a1a2-c881d271cd84/node-exporter/0.log" Sep 16 18:22:10.405226 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:10.405203 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzk4k_e6e500d3-c03a-4a33-a1a2-c881d271cd84/kube-rbac-proxy/0.log" Sep 16 18:22:10.447978 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:10.447958 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzk4k_e6e500d3-c03a-4a33-a1a2-c881d271cd84/init-textfile/0.log" Sep 16 18:22:13.313634 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.313600 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b7df64f45-jzlmd_8863c674-a354-4b62-8fab-0cb963cccab3/console/0.log" Sep 16 18:22:13.347193 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.347166 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-p5tkm_777ee835-a9c7-4eaa-b8ec-53db29935526/download-server/0.log" Sep 16 18:22:13.677810 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.677736 2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9"] Sep 16 18:22:13.678075 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678061 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678077 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678086 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="storage-initializer" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678092 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="storage-initializer" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678101 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678106 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678119 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.678123 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678124 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678132 2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kube-rbac-proxy" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678137 2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kube-rbac-proxy" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678191 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kube-rbac-proxy" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678201 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e29c91a-a2a0-4a3e-8c38-8347ae32e412" containerName="kserve-container" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678208 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.678333 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.678214 2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f568a1-0c02-435a-b068-6dbb8a616692" containerName="storage-initializer" Sep 16 18:22:13.681062 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.681042 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.683842 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.683816 2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hfg2l\"/\"default-dockercfg-gf7tn\"" Sep 16 18:22:13.683842 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.683827 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hfg2l\"/\"openshift-service-ca.crt\"" Sep 16 18:22:13.685170 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.685154 2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hfg2l\"/\"kube-root-ca.crt\"" Sep 16 18:22:13.688934 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.688893 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9"] Sep 16 18:22:13.777078 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.777048 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-sys\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.777261 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.777109 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt8h\" (UniqueName: \"kubernetes.io/projected/2bba12a7-a20e-4782-a662-8f311fd55b6a-kube-api-access-znt8h\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.777261 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.777164 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-podres\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.777261 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.777204 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-proc\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.777261 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.777227 2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-lib-modules\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877616 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877574 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt8h\" (UniqueName: \"kubernetes.io/projected/2bba12a7-a20e-4782-a662-8f311fd55b6a-kube-api-access-znt8h\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877624 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-podres\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877654 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-proc\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877705 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-lib-modules\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877748 2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-sys\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.877806 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877782 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-proc\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.878060 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877808 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-podres\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.878060 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877845 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-sys\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.878060 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.877894 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bba12a7-a20e-4782-a662-8f311fd55b6a-lib-modules\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.886366 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.886343 2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt8h\" (UniqueName: \"kubernetes.io/projected/2bba12a7-a20e-4782-a662-8f311fd55b6a-kube-api-access-znt8h\") pod \"perf-node-gather-daemonset-4tvw9\" (UID: \"2bba12a7-a20e-4782-a662-8f311fd55b6a\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:13.901778 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.901757 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-slxrd_dbd8fe4e-7065-417e-ac65-ab635b34da43/volume-data-source-validator/0.log" Sep 16 18:22:13.991644 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:13.991610 2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:14.260820 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:14.260793 2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9"] Sep 16 18:22:14.262720 ip-10-0-130-222 kubenswrapper[2536]: W0916 18:22:14.262691 2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2bba12a7_a20e_4782_a662_8f311fd55b6a.slice/crio-401c146b515431ce4946906602eb34d5bd2c6413fc514635483ab8ee77aa3cfb WatchSource:0}: Error finding container 401c146b515431ce4946906602eb34d5bd2c6413fc514635483ab8ee77aa3cfb: Status 404 returned error can't find the container with id 401c146b515431ce4946906602eb34d5bd2c6413fc514635483ab8ee77aa3cfb Sep 16 18:22:14.815821 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:14.815792 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5cxdl_c081013d-68e2-4d33-b424-f730a3aa56b3/dns/0.log" Sep 16 18:22:14.851780 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:14.851758 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5cxdl_c081013d-68e2-4d33-b424-f730a3aa56b3/kube-rbac-proxy/0.log" Sep 16 18:22:15.001238 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.001203 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" event={"ID":"2bba12a7-a20e-4782-a662-8f311fd55b6a","Type":"ContainerStarted","Data":"912169602cdd3220057e457ae9ac28282765b1daa1d776abe021e9f24868c659"} Sep 16 18:22:15.001420 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.001254 2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" event={"ID":"2bba12a7-a20e-4782-a662-8f311fd55b6a","Type":"ContainerStarted","Data":"401c146b515431ce4946906602eb34d5bd2c6413fc514635483ab8ee77aa3cfb"} Sep 16 18:22:15.001420 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.001325 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:15.037873 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.037815 2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" podStartSLOduration=2.037797859 podStartE2EDuration="2.037797859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:22:15.036590209 +0000 UTC m=+3442.979911828" watchObservedRunningTime="2026-09-16 18:22:15.037797859 +0000 UTC m=+3442.981119512" Sep 16 18:22:15.139699 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.139616 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5jxl_146dffd8-7263-4546-89e9-8a9b7dcc47fb/dns-node-resolver/0.log" Sep 16 18:22:15.579966 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.579932 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f499fd44b-rfrwj_2e601eaf-2fbe-40bb-91e1-376e6f635338/registry/0.log" Sep 16 18:22:15.648347 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:15.648323 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6qms_df81b8c0-eb31-498b-af78-a67a0cf5b8b0/node-ca/0.log" Sep 16 18:22:16.423282 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:16.423257 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bccd6f44-wl52g_3d8d0ef0-3c08-4c17-b45e-fb58983cbeb2/router/0.log" Sep 16 18:22:16.792029 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:16.792000 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g6bng_9ece61df-cba2-413b-b4bd-ca48cbef29b4/serve-healthcheck-canary/0.log" Sep 16 18:22:17.235708 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:17.235681 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fsg5h_4d026f29-dab6-48ed-b64c-83dc0670f229/insights-operator/1.log" Sep 16 18:22:17.235887 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:17.235760 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fsg5h_4d026f29-dab6-48ed-b64c-83dc0670f229/insights-operator/0.log" Sep 16 18:22:17.448565 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:17.448538 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8nkj_ef79accd-f342-4457-9cf6-e203e7ea3f2c/kube-rbac-proxy/0.log" Sep 16 18:22:17.470294 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:17.470266 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8nkj_ef79accd-f342-4457-9cf6-e203e7ea3f2c/exporter/0.log" Sep 16 18:22:17.494249 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:17.494175 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v8nkj_ef79accd-f342-4457-9cf6-e203e7ea3f2c/extractor/0.log" Sep 16 18:22:19.931062 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:19.931031 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cc8594966-8h852_8ca68af4-c999-4734-9cf3-22856a3ed8d6/manager/0.log" Sep 16 18:22:19.979849 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:19.979821 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-8qkgz_2177b553-aa4c-4840-a95e-4fa934e7565d/server/0.log" Sep 16 18:22:20.213767 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:20.213686 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-ncmrp_2b3b1702-cee2-4fc9-978c-96e2f0984ec5/manager/0.log" Sep 16 18:22:20.423496 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:20.423467 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-w5wv9_4fee5c41-9fa1-4e79-bb11-05fba6118799/seaweedfs/0.log" Sep 16 18:22:20.452695 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:20.452670 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-cschg_257473ad-fae8-475a-857e-eaafcb9f77a0/seaweedfs-tls-custom/0.log" Sep 16 18:22:20.482744 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:20.482725 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-h59vg_a63a44b2-ed7e-44a3-bfa4-c994ea6ddf68/seaweedfs-tls-serving/0.log" Sep 16 18:22:21.015434 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:21.015409 2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-4tvw9" Sep 16 18:22:26.141715 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.141677 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8xqfp_586a3038-36d1-4c2f-8739-97338f3663ba/kube-multus/0.log" Sep 16 18:22:26.168114 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.168044 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/kube-multus-additional-cni-plugins/0.log" Sep 16 18:22:26.194192 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.194161 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/egress-router-binary-copy/0.log" Sep 16 18:22:26.215879 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.215856 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/cni-plugins/0.log" Sep 16 18:22:26.235958 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.235933 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/bond-cni-plugin/0.log" Sep 16 18:22:26.257002 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.256964 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/routeoverride-cni/0.log" Sep 16 18:22:26.279200 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.279174 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/whereabouts-cni-bincopy/0.log" Sep 16 18:22:26.305513 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.305488 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fm9qz_7e336a64-28bc-443a-9705-7a2b9da2ec59/whereabouts-cni/0.log" Sep 16 18:22:26.793640 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.793608 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxj45_4cae5e35-ada9-445e-b9a6-7876dee2d9ff/network-metrics-daemon/0.log" Sep 16 18:22:26.819649 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:26.819624 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxj45_4cae5e35-ada9-445e-b9a6-7876dee2d9ff/kube-rbac-proxy/0.log" Sep 16 18:22:27.807734 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.807710 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-controller/0.log" Sep 16 18:22:27.827964 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.827940 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/0.log" Sep 16 18:22:27.828095 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.828021 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovn-acl-logging/1.log" Sep 16 18:22:27.848817 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.848790 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/kube-rbac-proxy-node/0.log" Sep 16 18:22:27.874074 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.874050 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 18:22:27.897502 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.897479 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/northd/0.log" Sep 16 18:22:27.933199 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.933172 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/nbdb/0.log" Sep 16 18:22:27.957093 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:27.957067 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/sbdb/0.log" Sep 16 18:22:28.000118 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:28.000086 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5ss65_2b5cbe64-815c-4b5f-91e5-2e90eb6939c0/ovnkube-controller/0.log" Sep 16 18:22:29.282106 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:29.282079 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b9pjl_e63f5ebe-c8e9-4538-b46d-85d6b3eff845/network-check-target-container/0.log" Sep 16 18:22:30.257041 ip-10-0-130-222 kubenswrapper[2536]: I0916 18:22:30.257012 2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xnm8r_f33bb888-0d54-4843-99a6-f88c642ef5aa/iptables-alerter/0.log"