May 25 11:15:15.606358 ip-10-0-141-33 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory May 25 11:15:15.606369 ip-10-0-141-33 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory May 25 11:15:15.606376 ip-10-0-141-33 systemd[1]: kubelet.service: Failed with result 'resources'. May 25 11:15:15.606594 ip-10-0-141-33 systemd[1]: Failed to start Kubernetes Kubelet. May 25 11:15:25.726890 ip-10-0-141-33 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. May 25 11:15:25.726918 ip-10-0-141-33 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot de76129b12eb4b2da2284cc75b499eb7 -- May 25 11:17:39.853397 ip-10-0-141-33 systemd[1]: Starting Kubernetes Kubelet... May 25 11:17:40.351224 ip-10-0-141-33 kubenswrapper[2548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 11:17:40.351224 ip-10-0-141-33 kubenswrapper[2548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 25 11:17:40.351224 ip-10-0-141-33 kubenswrapper[2548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 11:17:40.351224 ip-10-0-141-33 kubenswrapper[2548]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 25 11:17:40.351224 ip-10-0-141-33 kubenswrapper[2548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 11:17:40.354225 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.354143 2548 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 25 11:17:40.357155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357139 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 11:17:40.357155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357153 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 11:17:40.357155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357157 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357160 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357163 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357166 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357168 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357171 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357174 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357177 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357180 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357183 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357186 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357189 2548 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357191 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357199 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357202 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357205 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357207 2548 feature_gate.go:328] unrecognized feature gate: Example May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357210 2548 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357213 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357216 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 11:17:40.357261 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357218 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357223 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357226 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357229 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357232 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357234 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357237 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357240 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357243 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357245 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357247 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357250 2548 feature_gate.go:328] unrecognized feature gate: Example2 May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357252 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357255 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357258 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357261 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357265 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357268 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357271 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 11:17:40.357729 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357273 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357276 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357279 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357281 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357284 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357286 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357289 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357291 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357294 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357296 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357298 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357301 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357303 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357306 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357308 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357310 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357313 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357315 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357318 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357320 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 11:17:40.358205 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357323 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357325 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357328 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357330 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357333 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357335 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357338 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357341 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357344 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357347 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357350 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357353 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357355 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357357 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357361 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357365 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357368 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357371 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357373 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357376 2548 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 11:17:40.358681 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357378 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357380 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357383 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357385 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357388 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357746 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357751 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357754 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357756 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357759 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357761 2548 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357764 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357766 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357769 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357771 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357774 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357776 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357778 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357782 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357785 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 11:17:40.359174 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357788 2548 feature_gate.go:328] unrecognized feature gate: Example May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357790 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357801 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357804 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357806 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357809 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357811 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357814 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357816 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357819 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357821 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357824 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357826 2548 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357829 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357831 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357834 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357836 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357838 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357841 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357843 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 11:17:40.359814 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357846 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357848 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357850 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357853 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357855 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357858 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357860 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357862 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357865 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357867 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357870 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357874 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357877 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357880 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357884 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357888 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357891 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357895 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357898 2548 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 11:17:40.360558 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357901 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357903 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357906 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357909 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357911 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357914 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357917 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357919 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357922 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357924 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357927 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357929 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357931 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357934 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357936 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357939 2548 feature_gate.go:328] unrecognized feature gate: Example2 May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357941 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357943 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357946 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357948 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 11:17:40.361021 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357950 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357953 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357955 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357958 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357960 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357963 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357966 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357969 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357971 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357974 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357976 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.357979 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358742 2548 flags.go:64] FLAG: --address="0.0.0.0" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358752 2548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358759 2548 flags.go:64] FLAG: --anonymous-auth="true" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358763 2548 flags.go:64] FLAG: --application-metrics-count-limit="100" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358768 2548 flags.go:64] FLAG: --authentication-token-webhook="false" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358771 2548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358775 2548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358780 2548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358783 2548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 25 11:17:40.361560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358787 2548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358790 2548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358793 2548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358796 2548 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358799 2548 flags.go:64] FLAG: --cgroup-root="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358802 2548 flags.go:64] FLAG: --cgroups-per-qos="true" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358805 2548 flags.go:64] FLAG: --client-ca-file="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358808 2548 flags.go:64] FLAG: --cloud-config="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358811 2548 flags.go:64] FLAG: --cloud-provider="external" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358814 2548 flags.go:64] FLAG: --cluster-dns="[]" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358817 2548 flags.go:64] FLAG: --cluster-domain="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358820 2548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358823 2548 flags.go:64] FLAG: --config-dir="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358826 2548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358829 2548 flags.go:64] FLAG: --container-log-max-files="5" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358833 2548 flags.go:64] FLAG: --container-log-max-size="10Mi" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358836 2548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358839 2548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358843 2548 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358845 2548 flags.go:64] FLAG: --contention-profiling="false" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358848 2548 flags.go:64] FLAG: --cpu-cfs-quota="true" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358852 2548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358855 2548 flags.go:64] FLAG: --cpu-manager-policy="none" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358858 2548 flags.go:64] FLAG: --cpu-manager-policy-options="" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358862 2548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 25 11:17:40.362072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358865 2548 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358868 2548 flags.go:64] FLAG: --enable-debugging-handlers="true" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358871 2548 flags.go:64] FLAG: --enable-load-reader="false" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358874 2548 flags.go:64] FLAG: --enable-server="true" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358876 2548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358880 2548 flags.go:64] FLAG: --event-burst="100" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358883 2548 flags.go:64] FLAG: --event-qps="50" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358886 2548 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358889 2548 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358892 2548 flags.go:64] FLAG: --eviction-hard="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358896 2548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358899 2548 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358902 2548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358905 2548 flags.go:64] FLAG: --eviction-soft="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358908 2548 flags.go:64] FLAG: --eviction-soft-grace-period="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358911 2548 flags.go:64] FLAG: --exit-on-lock-contention="false" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358914 2548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358917 2548 flags.go:64] FLAG: --experimental-mounter-path="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358920 2548 flags.go:64] FLAG: --fail-cgroupv1="false" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358923 2548 flags.go:64] FLAG: --fail-swap-on="true" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358926 2548 flags.go:64] FLAG: --feature-gates="" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358930 2548 flags.go:64] FLAG: --file-check-frequency="20s" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358933 2548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358936 2548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358939 2548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358942 2548 flags.go:64] FLAG: --healthz-port="10248" May 25 11:17:40.362686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358945 2548 flags.go:64] FLAG: --help="false" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358948 2548 flags.go:64] FLAG: --hostname-override="ip-10-0-141-33.ec2.internal" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358951 2548 flags.go:64] FLAG: --housekeeping-interval="10s" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358955 2548 flags.go:64] FLAG: --http-check-frequency="20s" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358958 2548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358961 2548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358964 2548 flags.go:64] FLAG: --image-gc-high-threshold="85" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358967 2548 flags.go:64] FLAG: --image-gc-low-threshold="80" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358970 2548 flags.go:64] FLAG: --image-service-endpoint="" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358973 2548 flags.go:64] FLAG: --kernel-memcg-notification="false" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358976 2548 flags.go:64] FLAG: --kube-api-burst="100" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358978 2548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358981 2548 flags.go:64] FLAG: --kube-api-qps="50" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358984 2548 flags.go:64] FLAG: --kube-reserved="" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358987 2548 flags.go:64] FLAG: --kube-reserved-cgroup="" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358989 2548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358992 2548 flags.go:64] FLAG: --kubelet-cgroups="" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358995 2548 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.358998 2548 flags.go:64] FLAG: --lock-file="" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359004 2548 flags.go:64] FLAG: --log-cadvisor-usage="false" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359007 2548 flags.go:64] FLAG: --log-flush-frequency="5s" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359010 2548 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359015 2548 flags.go:64] FLAG: --log-json-split-stream="false" May 25 11:17:40.363338 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359018 2548 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359021 2548 flags.go:64] FLAG: --log-text-split-stream="false" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359024 2548 flags.go:64] FLAG: --logging-format="text" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359027 2548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359030 2548 flags.go:64] FLAG: --make-iptables-util-chains="true" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359033 2548 flags.go:64] FLAG: --manifest-url="" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359036 2548 flags.go:64] FLAG: --manifest-url-header="" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359040 2548 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359043 2548 flags.go:64] FLAG: --max-open-files="1000000" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359047 2548 flags.go:64] FLAG: --max-pods="110" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359050 2548 flags.go:64] FLAG: --maximum-dead-containers="-1" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359053 2548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359056 2548 flags.go:64] FLAG: --memory-manager-policy="None" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359060 2548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359063 2548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359066 2548 flags.go:64] FLAG: --node-ip="0.0.0.0" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359069 2548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359076 2548 flags.go:64] FLAG: --node-status-max-images="50" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359079 2548 flags.go:64] FLAG: --node-status-update-frequency="10s" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359082 2548 flags.go:64] FLAG: --oom-score-adj="-999" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359085 2548 flags.go:64] FLAG: --pod-cidr="" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359087 2548 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359092 2548 flags.go:64] FLAG: --pod-manifest-path="" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359108 2548 flags.go:64] FLAG: --pod-max-pids="-1" May 25 11:17:40.363884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359112 2548 flags.go:64] FLAG: --pods-per-core="0" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359115 2548 flags.go:64] FLAG: --port="10250" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359118 2548 flags.go:64] FLAG: --protect-kernel-defaults="false" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359120 2548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0723ffd48386fb50e" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359123 2548 flags.go:64] FLAG: --qos-reserved="" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359127 2548 flags.go:64] FLAG: --read-only-port="10255" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359130 2548 flags.go:64] FLAG: --register-node="true" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359133 2548 flags.go:64] FLAG: --register-schedulable="true" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359137 2548 flags.go:64] FLAG: --register-with-taints="" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359140 2548 flags.go:64] FLAG: --registry-burst="10" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359145 2548 flags.go:64] FLAG: --registry-qps="5" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359148 2548 flags.go:64] FLAG: --reserved-cpus="" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359151 2548 flags.go:64] FLAG: --reserved-memory="" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359154 2548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359158 2548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359161 2548 flags.go:64] FLAG: --rotate-certificates="false" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359164 2548 flags.go:64] FLAG: --rotate-server-certificates="false" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359167 2548 flags.go:64] FLAG: --runonce="false" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359170 2548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359173 2548 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359175 2548 flags.go:64] FLAG: --seccomp-default="false" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359178 2548 flags.go:64] FLAG: --serialize-image-pulls="true" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359182 2548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359185 2548 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359188 2548 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359191 2548 flags.go:64] FLAG: --storage-driver-password="root" May 25 11:17:40.364482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359194 2548 flags.go:64] FLAG: --storage-driver-secure="false" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359197 2548 flags.go:64] FLAG: --storage-driver-table="stats" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359202 2548 flags.go:64] FLAG: --storage-driver-user="root" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359204 2548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359207 2548 flags.go:64] FLAG: --sync-frequency="1m0s" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359210 2548 flags.go:64] FLAG: --system-cgroups="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359213 2548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359219 2548 flags.go:64] FLAG: --system-reserved-cgroup="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359221 2548 flags.go:64] FLAG: --tls-cert-file="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359224 2548 flags.go:64] FLAG: --tls-cipher-suites="[]" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359227 2548 flags.go:64] FLAG: --tls-min-version="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359231 2548 flags.go:64] FLAG: --tls-private-key-file="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359233 2548 flags.go:64] FLAG: --topology-manager-policy="none" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359236 2548 flags.go:64] FLAG: --topology-manager-policy-options="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359239 2548 flags.go:64] FLAG: --topology-manager-scope="container" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359242 2548 flags.go:64] FLAG: --v="2" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359247 2548 flags.go:64] FLAG: --version="false" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359253 2548 flags.go:64] FLAG: --vmodule="" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359257 2548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.359261 2548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361649 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361664 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361668 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361672 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 11:17:40.365155 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361674 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361677 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361680 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361683 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361685 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361688 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361690 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361693 2548 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361696 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361698 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361701 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361703 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361706 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361708 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361711 2548 feature_gate.go:328] unrecognized feature gate: Example2 May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361713 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361716 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361719 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361721 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361724 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 11:17:40.365736 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361727 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361730 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361732 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361735 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361737 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361740 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361743 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361745 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361747 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361755 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361758 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361760 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361763 2548 feature_gate.go:328] unrecognized feature gate: Example May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361766 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361768 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361771 2548 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361774 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361776 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361779 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361781 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 11:17:40.366244 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361784 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361786 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361788 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361791 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361793 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361796 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361798 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361801 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361804 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361806 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361808 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361811 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361814 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361818 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361822 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361824 2548 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361827 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361830 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361832 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 11:17:40.366730 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361835 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361837 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361842 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361844 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361847 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361850 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361852 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361854 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361857 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361860 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361862 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361866 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361870 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361872 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361875 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361878 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361881 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361884 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361886 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 11:17:40.367209 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361889 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 11:17:40.367667 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361891 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 11:17:40.367667 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361894 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 11:17:40.367667 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.361896 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 11:17:40.367667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.362785 2548 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 11:17:40.369655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.369637 2548 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 25 11:17:40.369655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.369655 2548 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369704 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369709 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369712 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369715 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369718 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369721 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369724 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369726 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369729 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 11:17:40.369727 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369732 2548 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369735 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369738 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369740 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369743 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369745 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369748 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369750 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369753 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369756 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369758 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369760 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369764 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369766 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369769 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369772 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369774 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369777 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369779 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369781 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 11:17:40.369972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369784 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369786 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369789 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369791 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369794 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369796 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369799 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369801 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369805 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369808 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369812 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369814 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369817 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369819 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369822 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369825 2548 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369828 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369831 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369833 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 11:17:40.370577 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369836 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369838 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369841 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369843 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369846 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369849 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369852 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369854 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369857 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369859 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369862 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369864 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369867 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369870 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369872 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369874 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369877 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369880 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369882 2548 feature_gate.go:328] unrecognized feature gate: Example May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369884 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 11:17:40.371047 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369887 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369889 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369892 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369895 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369897 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369900 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369902 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369907 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369911 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369914 2548 feature_gate.go:328] unrecognized feature gate: Example2 May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369917 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369920 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369923 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369925 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369928 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369930 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369933 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 11:17:40.371551 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.369935 2548 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.369941 2548 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370033 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370037 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370041 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370046 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370049 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370052 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370054 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370057 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370060 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370062 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370065 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370068 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370070 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 11:17:40.371974 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370073 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370076 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370079 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370082 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370084 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370087 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370089 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370092 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370112 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370117 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370119 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370122 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370125 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370127 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370130 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370133 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370135 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370138 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370141 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370143 2548 feature_gate.go:328] unrecognized feature gate: Example May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370147 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 11:17:40.372360 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370149 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370152 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370154 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370157 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370159 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370162 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370164 2548 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370166 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370169 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370172 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370191 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370195 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370198 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370201 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370204 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370206 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370208 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370211 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370213 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 11:17:40.372873 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370218 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370221 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370224 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370227 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370229 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370232 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370235 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370238 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370240 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370243 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370245 2548 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370248 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370250 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370253 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370256 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370258 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370261 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370263 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370265 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 11:17:40.373357 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370268 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370270 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370272 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370275 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370277 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370280 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370282 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370284 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370287 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370289 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370292 2548 feature_gate.go:328] unrecognized feature gate: Example2 May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370294 2548 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370297 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:40.370299 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.370304 2548 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 11:17:40.373823 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.371152 2548 server.go:962] "Client rotation is on, will bootstrap in background" May 25 11:17:40.377860 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.377846 2548 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 25 11:17:40.378876 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.378865 2548 server.go:1019] "Starting client certificate rotation" May 25 11:17:40.378980 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.378964 2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 25 11:17:40.379009 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.379003 2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 25 11:17:40.409060 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.409045 2548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 25 11:17:40.410186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.410169 2548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 25 11:17:40.430380 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.430362 2548 log.go:25] "Validated CRI v1 runtime API" May 25 11:17:40.438385 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.438366 2548 log.go:25] "Validated CRI v1 image API" May 25 11:17:40.441425 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.441409 2548 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 25 11:17:40.445795 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.445776 2548 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d25610ec-a4c0-4016-bed5-7465c8b977ad:/dev/nvme0n1p3 dd488f00-0f99-46f6-aee3-98bcba31624b:/dev/nvme0n1p4] May 25 11:17:40.445854 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.445796 2548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 25 11:17:40.451003 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.450762 2548 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:40.450282772 +0000 UTC m=+0.461891218 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097919 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b895cf15d12c7411d0568aa0e0697 SystemUUID:ec2b895c-f15d-12c7-411d-0568aa0e0697 BootID:de76129b-12eb-4b2d-a228-4cc75b499eb7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d4:0b:8a:22:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d4:0b:8a:22:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:d4:58:c0:3f:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 25 11:17:40.452701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.452689 2548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 25 11:17:40.452822 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.452809 2548 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 25 11:17:40.453470 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.453455 2548 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 25 11:17:40.454882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.454852 2548 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 25 11:17:40.455006 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.454884 2548 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-33.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 25 11:17:40.455056 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.455016 2548 topology_manager.go:138] "Creating topology manager with none policy" May 25 11:17:40.455056 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.455024 2548 container_manager_linux.go:306] "Creating device plugin manager" May 25 11:17:40.455056 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.455031 2548 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 25 11:17:40.455893 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.455882 2548 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 25 11:17:40.457884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.457873 2548 state_mem.go:36] "Initialized new in-memory state store" May 25 11:17:40.457991 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.457983 2548 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 25 11:17:40.461387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.461374 2548 kubelet.go:491] "Attempting to sync node with API server" May 25 11:17:40.461437 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.461390 2548 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 25 11:17:40.461437 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.461409 2548 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 25 11:17:40.461437 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.461419 2548 kubelet.go:397] "Adding apiserver pod source" May 25 11:17:40.461437 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.461426 2548 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 25 11:17:40.462596 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.462585 2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 25 11:17:40.462632 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.462603 2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 25 11:17:40.465771 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.465757 2548 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 25 11:17:40.468000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.467982 2548 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 25 11:17:40.470236 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470221 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 25 11:17:40.470236 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470237 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470244 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470250 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470258 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470263 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470271 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470276 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470283 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470288 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470304 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 25 11:17:40.470369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.470310 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 25 11:17:40.471400 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.471387 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 25 11:17:40.471447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.471403 2548 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 25 11:17:40.473570 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.473548 2548 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 25 11:17:40.473570 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.473548 2548 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 25 11:17:40.474518 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.474503 2548 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 25 11:17:40.474563 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.474548 2548 server.go:1295] "Started kubelet" May 25 11:17:40.474615 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.474600 2548 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-33.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 25 11:17:40.474686 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.474623 2548 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 25 11:17:40.474911 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.474828 2548 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 25 11:17:40.475016 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.475006 2548 server_v1.go:47] "podresources" method="list" useActivePods=true May 25 11:17:40.475338 ip-10-0-141-33 systemd[1]: Started Kubernetes Kubelet. May 25 11:17:40.476547 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.476516 2548 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 25 11:17:40.476882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.476867 2548 server.go:317] "Adding debug handlers to kubelet server" May 25 11:17:40.484348 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.484329 2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 25 11:17:40.484987 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.484971 2548 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 25 11:17:40.485144 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.485127 2548 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 25 11:17:40.485231 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.485176 2548 volume_manager.go:295] "The desired_state_of_world populator starts" May 25 11:17:40.485231 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.485189 2548 volume_manager.go:297] "Starting Kubelet Volume Manager" May 25 11:17:40.485329 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.485239 2548 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 25 11:17:40.485434 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.485420 2548 reconstruct.go:97] "Volume reconstruction finished" May 25 11:17:40.485434 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.485432 2548 reconciler.go:26] "Reconciler: start to sync state" May 25 11:17:40.485995 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.485977 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.486266 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486247 2548 factory.go:55] Registering systemd factory May 25 11:17:40.486360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486301 2548 factory.go:223] Registration of the systemd container factory successfully May 25 11:17:40.486523 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486511 2548 factory.go:153] Registering CRI-O factory May 25 11:17:40.486588 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486525 2548 factory.go:223] Registration of the crio container factory successfully May 25 11:17:40.486588 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486564 2548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 25 11:17:40.486588 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486583 2548 factory.go:103] Registering Raw factory May 25 11:17:40.486710 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.486593 2548 manager.go:1196] Started watching for new ooms in manager May 25 11:17:40.490451 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.490392 2548 manager.go:319] Starting recovery of all containers May 25 11:17:40.490852 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.490826 2548 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 25 11:17:40.491140 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.491116 2548 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 25 11:17:40.492547 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.491354 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-33.ec2.internal.18b2cadeab8ddaf1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-33.ec2.internal,UID:ip-10-0-141-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-33.ec2.internal,},FirstTimestamp:2026-05-25 11:17:40.474518257 +0000 UTC m=+0.486126709,LastTimestamp:2026-05-25 11:17:40.474518257 +0000 UTC m=+0.486126709,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-33.ec2.internal,}" May 25 11:17:40.497842 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.497805 2548 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 25 11:17:40.503448 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.503425 2548 manager.go:324] Recovery completed May 25 11:17:40.508149 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.508136 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.510398 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510380 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.510475 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510412 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.510475 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510426 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.510932 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510917 2548 cpu_manager.go:222] "Starting CPU manager" policy="none" May 25 11:17:40.510932 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510931 2548 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 25 11:17:40.511044 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.510948 2548 state_mem.go:36] "Initialized new in-memory state store" May 25 11:17:40.512702 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.512639 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-33.ec2.internal.18b2cadeadb151d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-33.ec2.internal,UID:ip-10-0-141-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-33.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-33.ec2.internal,},FirstTimestamp:2026-05-25 11:17:40.510396883 +0000 UTC m=+0.522005335,LastTimestamp:2026-05-25 11:17:40.510396883 +0000 UTC m=+0.522005335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-33.ec2.internal,}" May 25 11:17:40.514967 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.514954 2548 policy_none.go:49] "None policy: Start" May 25 11:17:40.515016 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.514973 2548 memory_manager.go:186] "Starting memorymanager" policy="None" May 25 11:17:40.515016 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.514983 2548 state_mem.go:35] "Initializing new in-memory state store" May 25 11:17:40.522931 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.522866 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-33.ec2.internal.18b2cadeadb1a753 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-33.ec2.internal,UID:ip-10-0-141-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-33.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-33.ec2.internal,},FirstTimestamp:2026-05-25 11:17:40.510418771 +0000 UTC m=+0.522027219,LastTimestamp:2026-05-25 11:17:40.510418771 +0000 UTC m=+0.522027219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-33.ec2.internal,}" May 25 11:17:40.532937 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.532860 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-33.ec2.internal.18b2cadeadb1d9b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-33.ec2.internal,UID:ip-10-0-141-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-33.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-33.ec2.internal,},FirstTimestamp:2026-05-25 11:17:40.510431666 +0000 UTC m=+0.522040113,LastTimestamp:2026-05-25 11:17:40.510431666 +0000 UTC m=+0.522040113,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-33.ec2.internal,}" May 25 11:17:40.553701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.553676 2548 manager.go:341] "Starting Device Plugin manager" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.553728 2548 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.553737 2548 server.go:85] "Starting device plugin registration server" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.554067 2548 eviction_manager.go:189] "Eviction manager: starting control loop" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.554083 2548 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.554196 2548 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.554271 2548 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.554280 2548 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.554693 2548 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 25 11:17:40.572568 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.554728 2548 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.583546 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.583480 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-33.ec2.internal.18b2cadeb108b545 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-33.ec2.internal,UID:ip-10-0-141-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-33.ec2.internal,},FirstTimestamp:2026-05-25 11:17:40.566455621 +0000 UTC m=+0.578064057,LastTimestamp:2026-05-25 11:17:40.566455621 +0000 UTC m=+0.578064057,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-33.ec2.internal,}" May 25 11:17:40.584529 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.584514 2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mkc22" May 25 11:17:40.591465 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.591451 2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mkc22" May 25 11:17:40.594301 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.594276 2548 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 25 11:17:40.594301 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.594299 2548 status_manager.go:230] "Starting to sync pod status with apiserver" May 25 11:17:40.594429 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.594318 2548 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 25 11:17:40.594429 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.594328 2548 kubelet.go:2451] "Starting kubelet main sync loop" May 25 11:17:40.594429 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.594369 2548 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 25 11:17:40.603560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.603521 2548 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 25 11:17:40.654722 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.654704 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.655482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.655466 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.655569 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.655500 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.655569 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.655515 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.655569 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.655542 2548 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-33.ec2.internal" May 25 11:17:40.664928 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.664913 2548 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-33.ec2.internal" May 25 11:17:40.664999 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.664933 2548 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-33.ec2.internal\": node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.686826 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.686804 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.695025 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.695007 2548 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal"] May 25 11:17:40.695081 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.695066 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.696462 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.696448 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.696539 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.696473 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.696539 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.696491 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.697479 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.697467 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.697634 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.697620 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.697683 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.697647 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.698175 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698160 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.698175 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698169 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.698279 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698192 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.698279 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698206 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.698279 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698192 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.698378 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.698303 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.699088 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.699075 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:40.699163 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.699119 2548 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 11:17:40.699791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.699775 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientMemory" May 25 11:17:40.699857 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.699806 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasNoDiskPressure" May 25 11:17:40.699857 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.699817 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeHasSufficientPID" May 25 11:17:40.723758 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.723741 2548 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-33.ec2.internal\" not found" node="ip-10-0-141-33.ec2.internal" May 25 11:17:40.728668 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.728652 2548 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-33.ec2.internal\" not found" node="ip-10-0-141-33.ec2.internal" May 25 11:17:40.786560 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.786540 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.786658 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.786566 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.786658 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.786581 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b41372adabb94dc7bf44dd30836c4051-config\") pod \"kube-apiserver-proxy-ip-10-0-141-33.ec2.internal\" (UID: \"b41372adabb94dc7bf44dd30836c4051\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:40.786964 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.786941 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.887424 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.887397 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:40.887567 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887537 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.888000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887805 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.888000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887866 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b41372adabb94dc7bf44dd30836c4051-config\") pod \"kube-apiserver-proxy-ip-10-0-141-33.ec2.internal\" (UID: \"b41372adabb94dc7bf44dd30836c4051\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:40.888000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887924 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.888000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887968 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fa2c7f43f2ddac74c4ae7d8975a3e10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal\" (UID: \"1fa2c7f43f2ddac74c4ae7d8975a3e10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:40.888000 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:40.887981 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b41372adabb94dc7bf44dd30836c4051-config\") pod \"kube-apiserver-proxy-ip-10-0-141-33.ec2.internal\" (UID: \"b41372adabb94dc7bf44dd30836c4051\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:40.987957 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:40.987927 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.026158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.026136 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:41.031270 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.031251 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:41.088330 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.088307 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.188703 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.188643 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.289038 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.289014 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.378368 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.378331 2548 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 25 11:17:41.379050 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.378546 2548 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 25 11:17:41.389666 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.389631 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.485222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.485195 2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 25 11:17:41.490322 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.490300 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.510190 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.510167 2548 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 25 11:17:41.534306 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.534284 2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2nmbd" May 25 11:17:41.544022 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.544001 2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2nmbd" May 25 11:17:41.590673 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.590644 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.595907 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.595877 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:40 +0000 UTC" deadline="2028-01-28 09:43:56.524258183 +0000 UTC" May 25 11:17:41.595998 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.595930 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14710h26m14.928339036s" May 25 11:17:41.617421 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:41.617396 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41372adabb94dc7bf44dd30836c4051.slice/crio-9a66b98a8963cd698284feca9d2fc35b59423a0b8db5fe93b492951b3ea4c818 WatchSource:0}: Error finding container 9a66b98a8963cd698284feca9d2fc35b59423a0b8db5fe93b492951b3ea4c818: Status 404 returned error can't find the container with id 9a66b98a8963cd698284feca9d2fc35b59423a0b8db5fe93b492951b3ea4c818 May 25 11:17:41.617660 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:41.617642 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa2c7f43f2ddac74c4ae7d8975a3e10.slice/crio-2d3de8dbac3905b0b12bca570da468dd0ec1549f41dc060a2e94af25291b149d WatchSource:0}: Error finding container 2d3de8dbac3905b0b12bca570da468dd0ec1549f41dc060a2e94af25291b149d: Status 404 returned error can't find the container with id 2d3de8dbac3905b0b12bca570da468dd0ec1549f41dc060a2e94af25291b149d May 25 11:17:41.620834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.620820 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 11:17:41.656504 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.656473 2548 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 25 11:17:41.690972 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:41.690948 2548 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-33.ec2.internal\" not found" May 25 11:17:41.711923 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.711900 2548 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 25 11:17:41.750034 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.749963 2548 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 25 11:17:41.786085 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.786056 2548 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" May 25 11:17:41.803845 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.803824 2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 25 11:17:41.805451 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.805438 2548 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" May 25 11:17:41.820937 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:41.820918 2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 25 11:17:42.437973 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.437943 2548 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 25 11:17:42.462198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.462137 2548 apiserver.go:52] "Watching apiserver" May 25 11:17:42.468552 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.468528 2548 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 25 11:17:42.469012 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.468986 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal","openshift-multus/multus-additional-cni-plugins-58hq2","openshift-multus/multus-rqdbv","openshift-network-operator/iptables-alerter-n6pld","openshift-ovn-kubernetes/ovnkube-node-tmtvc","kube-system/konnectivity-agent-mjfpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n","openshift-cluster-node-tuning-operator/tuned-rb5pn","openshift-multus/network-metrics-daemon-987s2","openshift-network-diagnostics/network-check-target-czj9c","kube-system/global-pull-secret-syncer-8wx4h","kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal","openshift-image-registry/node-ca-vwh8j"] May 25 11:17:42.471667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.471645 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.473807 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.473785 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.474387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.474368 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 25 11:17:42.474488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.474445 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kmbkd\"" May 25 11:17:42.474692 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.474673 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 25 11:17:42.474766 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.474740 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 25 11:17:42.475748 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.475730 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 25 11:17:42.476224 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.475967 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 25 11:17:42.476224 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.475994 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqdbv" May 25 11:17:42.476380 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.476363 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jcj7\"" May 25 11:17:42.476445 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.476396 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 25 11:17:42.477035 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.477021 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 25 11:17:42.477298 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.477280 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 25 11:17:42.478813 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.478129 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.478813 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.478174 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gszzw\"" May 25 11:17:42.478813 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.478474 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 25 11:17:42.482706 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.482679 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 25 11:17:42.482934 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.482913 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5r4b\"" May 25 11:17:42.483033 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.482941 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 25 11:17:42.483033 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.482913 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 25 11:17:42.485484 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.484967 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.485484 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.485002 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.487260 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487243 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 25 11:17:42.487347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487275 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5rv8b\"" May 25 11:17:42.487413 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487398 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 25 11:17:42.487493 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487478 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 25 11:17:42.487641 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487629 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 25 11:17:42.487736 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.487714 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 25 11:17:42.488337 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.488318 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 25 11:17:42.488427 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.488322 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hhwzg\"" May 25 11:17:42.488427 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.488404 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 25 11:17:42.488628 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.488553 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 25 11:17:42.489718 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.489700 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.489831 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.489814 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.489898 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.489877 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:42.491880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.491860 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 25 11:17:42.492093 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.492079 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bvsdf\"" May 25 11:17:42.492163 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.492143 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:42.492216 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.492195 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:42.492401 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.492381 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 25 11:17:42.494510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.494490 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.495039 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495022 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-bin\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495048 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-script-lib\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495066 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysconfig\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495091 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-netns\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495130 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz67\" (UniqueName: \"kubernetes.io/projected/1b22b242-6a7a-484a-ad28-64199fd493e0-kube-api-access-xmz67\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495145 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-sys\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495167 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-var-lib-kubelet\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495191 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-cnibin\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495213 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-bin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495234 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c92149ff-8de4-436e-95fe-05c9a447df6d-host-slash\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495253 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff79h\" (UniqueName: \"kubernetes.io/projected/c92149ff-8de4-436e-95fe-05c9a447df6d-kube-api-access-ff79h\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495272 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-k8s-cni-cncf-io\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495296 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-os-release\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495318 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495339 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-tmp\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495360 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-device-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.495386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495386 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-binary-copy\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495414 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-etc-kubernetes\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495452 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495477 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-kubernetes\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495512 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495535 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-multus-certs\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495557 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-env-overrides\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495579 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495618 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495674 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-node-log\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495698 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-modprobe-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495721 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-socket-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495749 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495794 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-system-cni-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495824 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-system-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495856 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.495940 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495884 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-netns\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495925 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-hostroot\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495952 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-sys-fs\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.495978 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496044 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b22b242-6a7a-484a-ad28-64199fd493e0-ovn-node-metrics-cert\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496076 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-agent-certs\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496121 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-konnectivity-ca\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496146 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-lib-modules\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496173 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cni-binary-copy\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496198 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-kubelet\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496222 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-slash\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496250 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-config\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496315 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdrw\" (UniqueName: \"kubernetes.io/projected/82daf91e-0f98-4b15-ba16-d1732291b935-kube-api-access-lsdrw\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496344 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-os-release\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496369 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-socket-dir-parent\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496391 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-daemon-config\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496413 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-ovn\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.496707 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496436 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-log-socket\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496460 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-netd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496493 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-run\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496523 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-registration-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496549 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6jx\" (UniqueName: \"kubernetes.io/projected/e2e48de0-b203-43d5-91b9-446157c1eada-kube-api-access-hn6jx\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496576 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-tuning-conf-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496599 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-systemd-units\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496625 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-var-lib-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496672 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-tuned\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496763 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-multus\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496791 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496815 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-conf\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496838 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cnibin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496862 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-kubelet\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496885 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-conf-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496909 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-systemd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.497458 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496934 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c92149ff-8de4-436e-95fe-05c9a447df6d-iptables-alerter-script\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496963 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-systemd\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496972 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.496987 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-host\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.497009 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94r5\" (UniqueName: \"kubernetes.io/projected/7a444b8d-a75f-49cb-916d-0be57c841be6-kube-api-access-q94r5\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.497021 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.497035 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968c5\" (UniqueName: \"kubernetes.io/projected/404ce4d8-0b4a-408e-9e3c-a45b682341b0-kube-api-access-968c5\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.497085 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-etc-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.497637 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 25 11:17:42.498212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.497633 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hw9xl\"" May 25 11:17:42.498611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.498239 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 25 11:17:42.498611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.498239 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 25 11:17:42.545242 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.545214 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:41 +0000 UTC" deadline="2027-12-30 03:35:11.751801326 +0000 UTC" May 25 11:17:42.545242 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.545239 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h17m29.206565019s" May 25 11:17:42.586495 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.586469 2548 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 25 11:17:42.597481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597453 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-conf-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.597609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597493 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-systemd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597518 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c92149ff-8de4-436e-95fe-05c9a447df6d-iptables-alerter-script\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.597609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597546 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-systemd\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597568 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-host\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597580 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-conf-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597589 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q94r5\" (UniqueName: \"kubernetes.io/projected/7a444b8d-a75f-49cb-916d-0be57c841be6-kube-api-access-q94r5\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597645 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-968c5\" (UniqueName: \"kubernetes.io/projected/404ce4d8-0b4a-408e-9e3c-a45b682341b0-kube-api-access-968c5\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597676 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-etc-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597703 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-bin\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597726 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-script-lib\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597750 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysconfig\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597774 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-netns\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597800 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz67\" (UniqueName: \"kubernetes.io/projected/1b22b242-6a7a-484a-ad28-64199fd493e0-kube-api-access-xmz67\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597824 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-sys\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597847 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-var-lib-kubelet\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.597880 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597886 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-cnibin\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597913 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-bin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597941 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9b0c54-3b7b-44e1-9614-0890e21e2157-host\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597965 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c92149ff-8de4-436e-95fe-05c9a447df6d-host-slash\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.597991 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ff79h\" (UniqueName: \"kubernetes.io/projected/c92149ff-8de4-436e-95fe-05c9a447df6d-kube-api-access-ff79h\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598016 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-k8s-cni-cncf-io\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598040 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-os-release\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598065 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598118 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-kubelet-config\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598187 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-tmp\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598212 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-device-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598245 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-binary-copy\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598269 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-etc-kubernetes\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598295 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598321 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-kubernetes\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598353 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598393 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-multus-certs\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.598419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598420 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-env-overrides\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598421 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c92149ff-8de4-436e-95fe-05c9a447df6d-iptables-alerter-script\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598454 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598481 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598505 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-node-log\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598508 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-systemd\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598543 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-dbus\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598560 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-host\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598570 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-modprobe-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598596 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-socket-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598629 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598654 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-system-cni-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598678 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-system-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598706 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598732 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-netns\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598758 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-hostroot\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598787 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-sys-fs\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.599186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598814 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598840 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b22b242-6a7a-484a-ad28-64199fd493e0-ovn-node-metrics-cert\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598867 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598894 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598953 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-agent-certs\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598980 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-konnectivity-ca\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.598983 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-etc-kubernetes\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599005 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-lib-modules\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599031 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cni-binary-copy\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599110 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-kubelet\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599141 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-slash\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599168 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-config\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599197 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599226 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdrw\" (UniqueName: \"kubernetes.io/projected/82daf91e-0f98-4b15-ba16-d1732291b935-kube-api-access-lsdrw\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599272 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-etc-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599319 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-os-release\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599349 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-socket-dir-parent\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.599979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599377 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-daemon-config\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599403 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-ovn\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599429 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-log-socket\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599454 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-netd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599484 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f9b0c54-3b7b-44e1-9614-0890e21e2157-serviceca\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599501 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599512 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-run\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599569 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-run\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599566 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-kubernetes\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599610 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-registration-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599642 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6jx\" (UniqueName: \"kubernetes.io/projected/e2e48de0-b203-43d5-91b9-446157c1eada-kube-api-access-hn6jx\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599685 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-tuning-conf-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599712 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-systemd-units\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599738 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-var-lib-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599769 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-tuned\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599800 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-multus\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599830 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5sqs\" (UniqueName: \"kubernetes.io/projected/2f9b0c54-3b7b-44e1-9614-0890e21e2157-kube-api-access-s5sqs\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.600834 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599856 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-kube-api-access-hr2k6\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599899 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599939 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-conf\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599971 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cnibin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.599998 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-kubelet\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600083 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-kubelet\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600206 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-script-lib\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600278 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysconfig\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600342 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-netns\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600611 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-sys\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600655 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-systemd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600680 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600708 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-var-lib-kubelet\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600756 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-multus-certs\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600758 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-bin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.600805 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c92149ff-8de4-436e-95fe-05c9a447df6d-host-slash\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601058 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-k8s-cni-cncf-io\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601126 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-env-overrides\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.601624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601161 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-os-release\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601190 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601209 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601503 2548 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601632 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601692 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-node-log\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601812 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-modprobe-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601925 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-socket-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.601984 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602035 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-system-cni-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602091 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-system-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602169 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-cni-dir\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.602212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602210 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-run-netns\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.602577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602249 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-hostroot\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.602577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602296 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-sys-fs\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.602577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602338 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602743 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-log-socket\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602782 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-netd\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602789 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-run-ovn\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602835 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-registration-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602849 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2e48de0-b203-43d5-91b9-446157c1eada-device-dir\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602871 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-os-release\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.602972 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-lib-modules\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603029 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-tuning-conf-dir\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603215 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-systemd-units\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603263 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-var-lib-openvswitch\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603291 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-daemon-config\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603358 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-multus-socket-dir-parent\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603448 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-host-var-lib-cni-multus\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.603575 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-d\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604078 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-slash\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604142 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-cni-bin\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604173 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b22b242-6a7a-484a-ad28-64199fd493e0-ovnkube-config\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.604593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604247 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cnibin\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604269 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82daf91e-0f98-4b15-ba16-d1732291b935-cni-binary-copy\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604273 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" event={"ID":"1fa2c7f43f2ddac74c4ae7d8975a3e10","Type":"ContainerStarted","Data":"2d3de8dbac3905b0b12bca570da468dd0ec1549f41dc060a2e94af25291b149d"} May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604368 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/404ce4d8-0b4a-408e-9e3c-a45b682341b0-cni-binary-copy\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604372 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-sysctl-conf\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604519 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82daf91e-0f98-4b15-ba16-d1732291b935-cnibin\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604624 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-tmp\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604715 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b22b242-6a7a-484a-ad28-64199fd493e0-host-kubelet\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.604981 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-konnectivity-ca\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.605510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.605153 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b22b242-6a7a-484a-ad28-64199fd493e0-ovn-node-metrics-cert\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.606194 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.606170 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad9ab7bf-e6d3-426c-b726-6c7e90cb1435-agent-certs\") pod \"konnectivity-agent-mjfpc\" (UID: \"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435\") " pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.606407 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.606386 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a444b8d-a75f-49cb-916d-0be57c841be6-etc-tuned\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.607011 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.606977 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" event={"ID":"b41372adabb94dc7bf44dd30836c4051","Type":"ContainerStarted","Data":"9a66b98a8963cd698284feca9d2fc35b59423a0b8db5fe93b492951b3ea4c818"} May 25 11:17:42.610736 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.610653 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdrw\" (UniqueName: \"kubernetes.io/projected/82daf91e-0f98-4b15-ba16-d1732291b935-kube-api-access-lsdrw\") pod \"multus-additional-cni-plugins-58hq2\" (UID: \"82daf91e-0f98-4b15-ba16-d1732291b935\") " pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.611188 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.611167 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94r5\" (UniqueName: \"kubernetes.io/projected/7a444b8d-a75f-49cb-916d-0be57c841be6-kube-api-access-q94r5\") pod \"tuned-rb5pn\" (UID: \"7a444b8d-a75f-49cb-916d-0be57c841be6\") " pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.611624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.611328 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff79h\" (UniqueName: \"kubernetes.io/projected/c92149ff-8de4-436e-95fe-05c9a447df6d-kube-api-access-ff79h\") pod \"iptables-alerter-n6pld\" (UID: \"c92149ff-8de4-436e-95fe-05c9a447df6d\") " pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.611624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.611549 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-968c5\" (UniqueName: \"kubernetes.io/projected/404ce4d8-0b4a-408e-9e3c-a45b682341b0-kube-api-access-968c5\") pod \"multus-rqdbv\" (UID: \"404ce4d8-0b4a-408e-9e3c-a45b682341b0\") " pod="openshift-multus/multus-rqdbv" May 25 11:17:42.612973 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.612954 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz67\" (UniqueName: \"kubernetes.io/projected/1b22b242-6a7a-484a-ad28-64199fd493e0-kube-api-access-xmz67\") pod \"ovnkube-node-tmtvc\" (UID: \"1b22b242-6a7a-484a-ad28-64199fd493e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.613094 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.613074 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6jx\" (UniqueName: \"kubernetes.io/projected/e2e48de0-b203-43d5-91b9-446157c1eada-kube-api-access-hn6jx\") pod \"aws-ebs-csi-driver-node-qtg9n\" (UID: \"e2e48de0-b203-43d5-91b9-446157c1eada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.701222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701154 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5sqs\" (UniqueName: \"kubernetes.io/projected/2f9b0c54-3b7b-44e1-9614-0890e21e2157-kube-api-access-s5sqs\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.701222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701184 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-kube-api-access-hr2k6\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.701222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701207 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9b0c54-3b7b-44e1-9614-0890e21e2157-host\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.701447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701262 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9b0c54-3b7b-44e1-9614-0890e21e2157-host\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.701447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701341 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-kubelet-config\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.701447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701394 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-dbus\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.701447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701425 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:42.701447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701432 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-kubelet-config\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701457 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701478 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701529 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f9b0c54-3b7b-44e1-9614-0890e21e2157-serviceca\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701561 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-dbus\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.701624 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:42.701704 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.701696 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:42.701966 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.701745 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:43.201720604 +0000 UTC m=+3.213329051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:42.701966 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.701766 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:43.201756171 +0000 UTC m=+3.213364607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:42.702047 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.701986 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f9b0c54-3b7b-44e1-9614-0890e21e2157-serviceca\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.708206 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.708176 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:42.708206 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.708208 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:42.708377 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.708221 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:42.708462 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:42.708443 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:43.208397933 +0000 UTC m=+3.220006380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:42.711801 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.711778 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-kube-api-access-hr2k6\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:42.711912 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.711827 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5sqs\" (UniqueName: \"kubernetes.io/projected/2f9b0c54-3b7b-44e1-9614-0890e21e2157-kube-api-access-s5sqs\") pod \"node-ca-vwh8j\" (UID: \"2f9b0c54-3b7b-44e1-9614-0890e21e2157\") " pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:42.785216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.785182 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" May 25 11:17:42.792048 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.792027 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-58hq2" May 25 11:17:42.801664 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.801645 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqdbv" May 25 11:17:42.807235 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.807219 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6pld" May 25 11:17:42.814210 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.814188 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:17:42.820764 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.820746 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:17:42.826927 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.826909 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" May 25 11:17:42.836389 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:42.836372 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwh8j" May 25 11:17:43.204797 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.204602 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:43.204797 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.204646 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:43.204797 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.204781 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:43.204797 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.204785 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:43.205002 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.204843 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.204824346 +0000 UTC m=+4.216432779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:43.205002 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.204864 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.204855408 +0000 UTC m=+4.216463842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:43.207297 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.207275 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a444b8d_a75f_49cb_916d_0be57c841be6.slice/crio-3df8437ce6567976c39f15ed91871226e4fb70f6a686acfddeecc10fbee0df2a WatchSource:0}: Error finding container 3df8437ce6567976c39f15ed91871226e4fb70f6a686acfddeecc10fbee0df2a: Status 404 returned error can't find the container with id 3df8437ce6567976c39f15ed91871226e4fb70f6a686acfddeecc10fbee0df2a May 25 11:17:43.208080 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.208057 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92149ff_8de4_436e_95fe_05c9a447df6d.slice/crio-e43a9af76a2d5366aa2a1993d9b819ed7892c2c9a304f36bb94c08ecc218af69 WatchSource:0}: Error finding container e43a9af76a2d5366aa2a1993d9b819ed7892c2c9a304f36bb94c08ecc218af69: Status 404 returned error can't find the container with id e43a9af76a2d5366aa2a1993d9b819ed7892c2c9a304f36bb94c08ecc218af69 May 25 11:17:43.210477 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.210453 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404ce4d8_0b4a_408e_9e3c_a45b682341b0.slice/crio-28dd1a684f2fa925743875fb5e99e98ca51c916125a2c5ced1dc9348aa98b288 WatchSource:0}: Error finding container 28dd1a684f2fa925743875fb5e99e98ca51c916125a2c5ced1dc9348aa98b288: Status 404 returned error can't find the container with id 28dd1a684f2fa925743875fb5e99e98ca51c916125a2c5ced1dc9348aa98b288 May 25 11:17:43.211948 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.211850 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82daf91e_0f98_4b15_ba16_d1732291b935.slice/crio-edf2169eaafb27727c7cedb4c66d6bfead5135d8c6754797a8f0e562b309994f WatchSource:0}: Error finding container edf2169eaafb27727c7cedb4c66d6bfead5135d8c6754797a8f0e562b309994f: Status 404 returned error can't find the container with id edf2169eaafb27727c7cedb4c66d6bfead5135d8c6754797a8f0e562b309994f May 25 11:17:43.212460 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.212435 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e48de0_b203_43d5_91b9_446157c1eada.slice/crio-6d03ce3926f32e2f88640b2033ecb2706a81051bc791daf7fc72f5600027f376 WatchSource:0}: Error finding container 6d03ce3926f32e2f88640b2033ecb2706a81051bc791daf7fc72f5600027f376: Status 404 returned error can't find the container with id 6d03ce3926f32e2f88640b2033ecb2706a81051bc791daf7fc72f5600027f376 May 25 11:17:43.217409 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:17:43.217339 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9ab7bf_e6d3_426c_b726_6c7e90cb1435.slice/crio-c57ee18c0b62c76e9fd97ef4163c720326a0c967b93ecd74c5e7057f2843b470 WatchSource:0}: Error finding container c57ee18c0b62c76e9fd97ef4163c720326a0c967b93ecd74c5e7057f2843b470: Status 404 returned error can't find the container with id c57ee18c0b62c76e9fd97ef4163c720326a0c967b93ecd74c5e7057f2843b470 May 25 11:17:43.305858 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.305836 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:43.306017 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.306000 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:43.306088 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.306020 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:43.306088 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.306035 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:43.306088 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.306079 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.306065648 +0000 UTC m=+4.317674091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:43.545911 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.545875 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:41 +0000 UTC" deadline="2028-02-22 17:29:29.763739716 +0000 UTC" May 25 11:17:43.546232 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.545912 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15318h11m46.217831578s" May 25 11:17:43.595960 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.595485 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:43.595960 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:43.595613 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:43.614709 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.614629 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" event={"ID":"e2e48de0-b203-43d5-91b9-446157c1eada","Type":"ContainerStarted","Data":"6d03ce3926f32e2f88640b2033ecb2706a81051bc791daf7fc72f5600027f376"} May 25 11:17:43.617258 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.617227 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqdbv" event={"ID":"404ce4d8-0b4a-408e-9e3c-a45b682341b0","Type":"ContainerStarted","Data":"28dd1a684f2fa925743875fb5e99e98ca51c916125a2c5ced1dc9348aa98b288"} May 25 11:17:43.619537 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.619513 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6pld" event={"ID":"c92149ff-8de4-436e-95fe-05c9a447df6d","Type":"ContainerStarted","Data":"e43a9af76a2d5366aa2a1993d9b819ed7892c2c9a304f36bb94c08ecc218af69"} May 25 11:17:43.629386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.629359 2548 generic.go:358] "Generic (PLEG): container finished" podID="1fa2c7f43f2ddac74c4ae7d8975a3e10" containerID="7c37d8f324305f04d12d07352ead08e6234afbb886b5947178ce91a346280fef" exitCode=0 May 25 11:17:43.629489 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.629431 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" event={"ID":"1fa2c7f43f2ddac74c4ae7d8975a3e10","Type":"ContainerDied","Data":"7c37d8f324305f04d12d07352ead08e6234afbb886b5947178ce91a346280fef"} May 25 11:17:43.634955 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.634929 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mjfpc" event={"ID":"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435","Type":"ContainerStarted","Data":"c57ee18c0b62c76e9fd97ef4163c720326a0c967b93ecd74c5e7057f2843b470"} May 25 11:17:43.636597 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.636573 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwh8j" event={"ID":"2f9b0c54-3b7b-44e1-9614-0890e21e2157","Type":"ContainerStarted","Data":"e7801f588d0f36fa78a2e1f9e24a18336de0ed0b8f4703355ec2623f528aaf89"} May 25 11:17:43.639331 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.639307 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerStarted","Data":"edf2169eaafb27727c7cedb4c66d6bfead5135d8c6754797a8f0e562b309994f"} May 25 11:17:43.642891 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.642822 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" event={"ID":"7a444b8d-a75f-49cb-916d-0be57c841be6","Type":"ContainerStarted","Data":"3df8437ce6567976c39f15ed91871226e4fb70f6a686acfddeecc10fbee0df2a"} May 25 11:17:43.655296 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.655251 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"976ef2b2607d87b7441063d0e7327e87dd1aa969a1852e58393e3c4509274072"} May 25 11:17:43.659619 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:43.659593 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" event={"ID":"b41372adabb94dc7bf44dd30836c4051","Type":"ContainerStarted","Data":"5c7510059f9dff721063dcf10882df986b883c19c2a5a2583e43ade01d460f59"} May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.213327 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.213374 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.213512 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.213571 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:46.213551301 +0000 UTC m=+6.225159741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.213933 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:44.217134 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.213977 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:46.213962743 +0000 UTC m=+6.225571176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:44.313998 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.313965 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:44.314204 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.314175 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:44.314204 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.314197 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:44.314328 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.314209 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:44.314328 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.314262 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:46.314245004 +0000 UTC m=+6.325853442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:44.596413 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.595826 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:44.596413 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.595968 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:44.603709 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.601813 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:44.603709 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:44.601990 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:44.671218 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.671125 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" event={"ID":"1fa2c7f43f2ddac74c4ae7d8975a3e10","Type":"ContainerStarted","Data":"82721c8e9951e5e5b1129ce5e42adaf4a61600070974da62293349c889ca8dfc"} May 25 11:17:44.685416 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.685365 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-33.ec2.internal" podStartSLOduration=3.6853491910000002 podStartE2EDuration="3.685349191s" podCreationTimestamp="2026-05-25 11:17:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:43.674006471 +0000 UTC m=+3.685614927" watchObservedRunningTime="2026-05-25 11:17:44.685349191 +0000 UTC m=+4.696957641" May 25 11:17:44.685551 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:44.685504 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-33.ec2.internal" podStartSLOduration=3.685496738 podStartE2EDuration="3.685496738s" podCreationTimestamp="2026-05-25 11:17:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:44.684333775 +0000 UTC m=+4.695942232" watchObservedRunningTime="2026-05-25 11:17:44.685496738 +0000 UTC m=+4.697105197" May 25 11:17:45.595074 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:45.595043 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:45.595257 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:45.595184 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:46.229829 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:46.229775 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:46.229829 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:46.229818 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:46.230366 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.229916 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:46.230366 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.229919 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:46.230366 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.229977 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.229956671 +0000 UTC m=+10.241565110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:46.230366 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.229995 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.229986746 +0000 UTC m=+10.241595184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:46.331136 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:46.330478 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:46.331136 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.330684 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:46.331136 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.330702 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:46.331136 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.330714 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:46.331136 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.330768 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.330749668 +0000 UTC m=+10.342358121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:46.596906 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:46.596253 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:46.596906 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.596429 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:46.597386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:46.597237 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:46.597386 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:46.597346 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:47.595453 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:47.595418 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:47.595882 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:47.595539 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:48.595194 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:48.595165 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:48.595360 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:48.595272 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:48.595585 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:48.595564 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:48.595936 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:48.595638 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:49.594831 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:49.594798 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:49.595093 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:49.594931 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:50.262842 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:50.262804 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:50.263387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:50.262862 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:50.276316 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.276281 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:50.276470 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.276390 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:58.276367354 +0000 UTC m=+18.287975790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:50.276967 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.276839 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:50.276967 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.276909 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:58.276890504 +0000 UTC m=+18.288498953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:50.363758 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:50.363713 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:50.363928 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.363910 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:50.363984 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.363935 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:50.363984 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.363949 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:50.364064 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.364004 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:58.363985781 +0000 UTC m=+18.375594220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:50.599255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:50.598539 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:50.599255 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.598645 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:50.599255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:50.599038 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:50.599255 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:50.599160 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:51.595114 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:51.595064 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:51.595552 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:51.595211 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:52.594819 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:52.594780 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:52.594819 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:52.594822 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:52.595066 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:52.594893 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:52.595066 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:52.595038 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:53.594827 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:53.594730 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:53.594969 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:53.594908 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:54.595299 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:54.595263 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:54.595299 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:54.595287 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:54.595745 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:54.595382 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:54.595745 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:54.595519 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:55.594727 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:55.594698 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:55.594894 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:55.594794 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:56.594690 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:56.594658 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:56.595065 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:56.594694 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:56.595065 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:56.594783 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:56.595065 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:56.594893 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:57.594925 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:57.594890 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:57.595336 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:57.595007 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:17:58.323112 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:58.323078 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:58.323290 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:58.323138 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:58.323290 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.323243 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:58.323290 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.323262 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:17:58.323400 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.323312 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.323292051 +0000 UTC m=+34.334900483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:17:58.323400 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.323332 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.323321342 +0000 UTC m=+34.334929775 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:17:58.423936 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:58.423904 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:58.424142 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.424060 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:17:58.424142 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.424085 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:17:58.424142 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.424114 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:58.424271 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.424172 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.424159114 +0000 UTC m=+34.435767556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:17:58.595239 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:58.595150 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:17:58.595661 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:58.595163 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:17:58.595661 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.595287 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:17:58.595661 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:58.595373 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:17:59.594881 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:17:59.594847 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:17:59.595053 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:17:59.594956 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:00.595724 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:00.595679 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:00.596302 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:00.595736 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:00.596302 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:00.595809 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:00.596302 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:00.595888 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:01.595475 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.595324 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:01.595576 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:01.595556 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:01.700452 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.700421 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"f7c673a667350aa1d50ffed8878750fcec58d43e0202a91b720c3dcebbb12a54"} May 25 11:18:01.701758 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.701730 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" event={"ID":"e2e48de0-b203-43d5-91b9-446157c1eada","Type":"ContainerStarted","Data":"7390443257ec227066879033af628c2938888b8295ac2a9c74849adf18db3fc8"} May 25 11:18:01.703491 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.703288 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqdbv" event={"ID":"404ce4d8-0b4a-408e-9e3c-a45b682341b0","Type":"ContainerStarted","Data":"ababd2e805480486add2670cfadbf3a1e4471d317b2360ad40552e427c89e478"} May 25 11:18:01.704487 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.704464 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mjfpc" event={"ID":"ad9ab7bf-e6d3-426c-b726-6c7e90cb1435","Type":"ContainerStarted","Data":"6ae193ce43762c27fdc84d79b08ddda379e16ee613a6bcdecc0f2504fcd41d02"} May 25 11:18:01.705593 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.705571 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwh8j" event={"ID":"2f9b0c54-3b7b-44e1-9614-0890e21e2157","Type":"ContainerStarted","Data":"c77c1ee818cf4cc0d6fd7a78acc7c7840e6764bfdda333996ed2c46e599e00b8"} May 25 11:18:01.706567 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.706548 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerStarted","Data":"37dffec9a0d23da48016b1a54c171c0fb7681b862bd3f9d9e23db47bdde13d60"} May 25 11:18:01.707737 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.707718 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" event={"ID":"7a444b8d-a75f-49cb-916d-0be57c841be6","Type":"ContainerStarted","Data":"7ca9e05c346603d7650f48b61cdc331b2579f5eed5bca57bcd120ca3a67911a6"} May 25 11:18:01.734541 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.734489 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-rqdbv" podStartSLOduration=3.556681603 podStartE2EDuration="21.73446918s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.212423219 +0000 UTC m=+3.224031669" lastFinishedPulling="2026-05-25 11:18:01.390210801 +0000 UTC m=+21.401819246" observedRunningTime="2026-05-25 11:18:01.719457429 +0000 UTC m=+21.731065882" watchObservedRunningTime="2026-05-25 11:18:01.73446918 +0000 UTC m=+21.746077637" May 25 11:18:01.735151 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.735087 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rb5pn" podStartSLOduration=3.570946546 podStartE2EDuration="21.735074956s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.209446516 +0000 UTC m=+3.221054950" lastFinishedPulling="2026-05-25 11:18:01.373574921 +0000 UTC m=+21.385183360" observedRunningTime="2026-05-25 11:18:01.734626457 +0000 UTC m=+21.746234911" watchObservedRunningTime="2026-05-25 11:18:01.735074956 +0000 UTC m=+21.746683412" May 25 11:18:01.775004 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:01.774813 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vwh8j" podStartSLOduration=3.641532592 podStartE2EDuration="21.774798216s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.218361454 +0000 UTC m=+3.229969887" lastFinishedPulling="2026-05-25 11:18:01.351627064 +0000 UTC m=+21.363235511" observedRunningTime="2026-05-25 11:18:01.755494376 +0000 UTC m=+21.767102831" watchObservedRunningTime="2026-05-25 11:18:01.774798216 +0000 UTC m=+21.786406671" May 25 11:18:02.595088 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.595065 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:02.595248 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.595134 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:02.595248 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:02.595225 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:02.595364 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:02.595305 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:02.682562 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.682542 2548 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 25 11:18:02.710603 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.710456 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="37dffec9a0d23da48016b1a54c171c0fb7681b862bd3f9d9e23db47bdde13d60" exitCode=0 May 25 11:18:02.710960 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.710541 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"37dffec9a0d23da48016b1a54c171c0fb7681b862bd3f9d9e23db47bdde13d60"} May 25 11:18:02.713294 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713273 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:18:02.713609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713589 2548 generic.go:358] "Generic (PLEG): container finished" podID="1b22b242-6a7a-484a-ad28-64199fd493e0" containerID="76dd4f854c5a61e53c2c9d00c3b6aec903d1f8144b49b745210363e972e7e4d3" exitCode=1 May 25 11:18:02.713708 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713655 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"32f16c6f83171041c61b32300d4a6cc819db75bce080afcce04a0b78af6ccb91"} May 25 11:18:02.713708 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713679 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"68ec0acc5e0934aa74e56da9f876a1e7398b46ad856c1d5feab6a3cc4b87c1b1"} May 25 11:18:02.713708 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713688 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"3aaf7cb9b0ff711caa155815006d2ed8e10a2ed7063b2a43db219df86d73f4b8"} May 25 11:18:02.713708 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713700 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"a1b31551bd52723588cea4b9a79496c688cff9a1e7d62a4eccf5d208927621e3"} May 25 11:18:02.713857 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.713713 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerDied","Data":"76dd4f854c5a61e53c2c9d00c3b6aec903d1f8144b49b745210363e972e7e4d3"} May 25 11:18:02.715262 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.715229 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" event={"ID":"e2e48de0-b203-43d5-91b9-446157c1eada","Type":"ContainerStarted","Data":"6dbd8933c6f738632cb16253173ea6d404656b5a6de2e1c592b2ce34bf46b2d2"} May 25 11:18:02.728923 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:02.728886 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mjfpc" podStartSLOduration=4.596736159 podStartE2EDuration="22.7288755s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.219488588 +0000 UTC m=+3.231097023" lastFinishedPulling="2026-05-25 11:18:01.351627917 +0000 UTC m=+21.363236364" observedRunningTime="2026-05-25 11:18:01.787876391 +0000 UTC m=+21.799484846" watchObservedRunningTime="2026-05-25 11:18:02.7288755 +0000 UTC m=+22.740483996" May 25 11:18:03.566904 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.566798 2548 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:18:02.682558637Z","UUID":"b7d98d09-7331-43e1-88e4-27cd0a5b8346","Handler":null,"Name":"","Endpoint":""} May 25 11:18:03.570323 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.570300 2548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 25 11:18:03.570444 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.570333 2548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 25 11:18:03.594629 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.594598 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:03.594778 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:03.594716 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:03.718863 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.718829 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6pld" event={"ID":"c92149ff-8de4-436e-95fe-05c9a447df6d","Type":"ContainerStarted","Data":"cf6ccc3ca7530a4a116172d7b17fd8fa37b7386d83640c0da4e61a0d23600d24"} May 25 11:18:03.737463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:03.737416 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n6pld" podStartSLOduration=5.573850323 podStartE2EDuration="23.737400927s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.210422922 +0000 UTC m=+3.222031370" lastFinishedPulling="2026-05-25 11:18:01.373973541 +0000 UTC m=+21.385581974" observedRunningTime="2026-05-25 11:18:03.736707747 +0000 UTC m=+23.748316202" watchObservedRunningTime="2026-05-25 11:18:03.737400927 +0000 UTC m=+23.749009382" May 25 11:18:04.594997 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.594912 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:04.594997 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.594939 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:04.595232 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:04.595073 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:04.595232 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:04.595201 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:04.725906 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.725877 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:18:04.726571 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.726229 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"4a6dad80c004a7ab52a4a149ec9fad5a87a32bd4690da7851e25f61f90440243"} May 25 11:18:04.727969 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.727940 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" event={"ID":"e2e48de0-b203-43d5-91b9-446157c1eada","Type":"ContainerStarted","Data":"e52fe81e5a959e42df1a94f8b098880f97249017c0329888caa8ab9ea4958b43"} May 25 11:18:04.747544 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:04.747492 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtg9n" podStartSLOduration=4.226678954 podStartE2EDuration="24.74747636s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.216500641 +0000 UTC m=+3.228109079" lastFinishedPulling="2026-05-25 11:18:03.737298052 +0000 UTC m=+23.748906485" observedRunningTime="2026-05-25 11:18:04.747475894 +0000 UTC m=+24.759084349" watchObservedRunningTime="2026-05-25 11:18:04.74747636 +0000 UTC m=+24.759084817" May 25 11:18:05.594800 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:05.594770 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:05.594958 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:05.594892 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:05.878136 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:05.878080 2548 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:18:05.878821 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:05.878800 2548 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:18:06.595498 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:06.595464 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:06.595664 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:06.595604 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:06.595721 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:06.595671 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:06.595820 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:06.595798 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:06.730863 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:06.730690 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:18:06.731154 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:06.731140 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mjfpc" May 25 11:18:07.595542 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.595509 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:07.595979 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:07.595615 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:07.733608 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.733571 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="2ae26550e934c997515d52fbac073a7adf49ce19c99d8c759aa2d30801702824" exitCode=0 May 25 11:18:07.733777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.733655 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"2ae26550e934c997515d52fbac073a7adf49ce19c99d8c759aa2d30801702824"} May 25 11:18:07.736996 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.736981 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:18:07.737449 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.737423 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"97bc544644a0f761897b4dd3c8ce6d688ed276eb381b223f65d43f8b0671b047"} May 25 11:18:07.737762 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.737738 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:07.737841 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.737792 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:07.737884 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.737854 2548 scope.go:117] "RemoveContainer" containerID="76dd4f854c5a61e53c2c9d00c3b6aec903d1f8144b49b745210363e972e7e4d3" May 25 11:18:07.754766 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:07.754744 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:08.595682 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.595487 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:08.596176 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.595500 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:08.596176 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:08.595792 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:08.596176 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:08.595864 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:08.679479 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.679445 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-987s2"] May 25 11:18:08.679617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.679494 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-czj9c"] May 25 11:18:08.679617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.679592 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:08.679721 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:08.679678 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:08.680892 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.680854 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8wx4h"] May 25 11:18:08.741804 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.741782 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:18:08.742185 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.742158 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" event={"ID":"1b22b242-6a7a-484a-ad28-64199fd493e0","Type":"ContainerStarted","Data":"81ae8453379bfc300fa033075ce3e5aa599445168ad87abd91180b7b6ed574a6"} May 25 11:18:08.742185 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.742174 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:08.742308 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.742209 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:08.742355 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:08.742332 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:08.742421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.742408 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:08.742481 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:08.742409 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:08.758150 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.758128 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:08.773830 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:08.773791 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" podStartSLOduration=10.558200862 podStartE2EDuration="28.773778679s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.206893601 +0000 UTC m=+3.218502036" lastFinishedPulling="2026-05-25 11:18:01.422471415 +0000 UTC m=+21.434079853" observedRunningTime="2026-05-25 11:18:08.772522227 +0000 UTC m=+28.784130682" watchObservedRunningTime="2026-05-25 11:18:08.773778679 +0000 UTC m=+28.785387133" May 25 11:18:09.745938 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:09.745903 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="6d652b85a8b6cc5794738de1e4f53bddc9de7365be006bf8642d42bb17217e5e" exitCode=0 May 25 11:18:09.746468 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:09.745996 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"6d652b85a8b6cc5794738de1e4f53bddc9de7365be006bf8642d42bb17217e5e"} May 25 11:18:10.595232 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:10.595202 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:10.595375 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:10.595248 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:10.595375 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:10.595322 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:10.595493 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:10.595380 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:10.595493 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:10.595435 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:10.595576 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:10.595524 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:11.752025 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:11.751989 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="8967af1a41247c4482fdb679443e2115b0226069511a7bbe89cac20b340b246f" exitCode=0 May 25 11:18:11.752623 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:11.752059 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"8967af1a41247c4482fdb679443e2115b0226069511a7bbe89cac20b340b246f"} May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.595295 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.595346 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:12.595420 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wx4h" podUID="79fe35ea-6771-4a9e-ab6f-3790e1c579a1" May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:12.595472 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czj9c" podUID="8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1" May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.595496 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:12.595824 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:12.595586 2548 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-987s2" podUID="8a17bca6-35d7-416a-ab02-2a1c52de2b5a" May 25 11:18:12.610175 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.610150 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-jf4zc"] May 25 11:18:12.630340 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.630319 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.633213 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.633052 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 25 11:18:12.633343 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.633307 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j6lnl\"" May 25 11:18:12.633343 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.633314 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 25 11:18:12.732758 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.732733 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d68c20-0bdc-4cc2-8357-885bba30014f-tmp-dir\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.732893 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.732773 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmx2\" (UniqueName: \"kubernetes.io/projected/06d68c20-0bdc-4cc2-8357-885bba30014f-kube-api-access-zpmx2\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.732893 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.732875 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d68c20-0bdc-4cc2-8357-885bba30014f-hosts-file\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.833529 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.833494 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d68c20-0bdc-4cc2-8357-885bba30014f-hosts-file\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.833910 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.833535 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d68c20-0bdc-4cc2-8357-885bba30014f-tmp-dir\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.833910 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.833586 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmx2\" (UniqueName: \"kubernetes.io/projected/06d68c20-0bdc-4cc2-8357-885bba30014f-kube-api-access-zpmx2\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.833910 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.833626 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d68c20-0bdc-4cc2-8357-885bba30014f-hosts-file\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.834086 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.833940 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d68c20-0bdc-4cc2-8357-885bba30014f-tmp-dir\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.845189 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.844996 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmx2\" (UniqueName: \"kubernetes.io/projected/06d68c20-0bdc-4cc2-8357-885bba30014f-kube-api-access-zpmx2\") pod \"node-resolver-jf4zc\" (UID: \"06d68c20-0bdc-4cc2-8357-885bba30014f\") " pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.942203 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:12.942175 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jf4zc" May 25 11:18:12.950799 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:12.950777 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d68c20_0bdc_4cc2_8357_885bba30014f.slice/crio-fca2eee8ed2334d767d16e3f100afe3dedc3f07e567db57cc11f2fc7b4363928 WatchSource:0}: Error finding container fca2eee8ed2334d767d16e3f100afe3dedc3f07e567db57cc11f2fc7b4363928: Status 404 returned error can't find the container with id fca2eee8ed2334d767d16e3f100afe3dedc3f07e567db57cc11f2fc7b4363928 May 25 11:18:13.758735 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:13.758694 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jf4zc" event={"ID":"06d68c20-0bdc-4cc2-8357-885bba30014f","Type":"ContainerStarted","Data":"f5332401e680e6055d5e4447b6c9cd832630f7653cfba6c360ec37a8e0aa7c06"} May 25 11:18:13.758735 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:13.758738 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jf4zc" event={"ID":"06d68c20-0bdc-4cc2-8357-885bba30014f","Type":"ContainerStarted","Data":"fca2eee8ed2334d767d16e3f100afe3dedc3f07e567db57cc11f2fc7b4363928"} May 25 11:18:13.774683 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:13.774633 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-jf4zc" podStartSLOduration=1.7746193959999998 podStartE2EDuration="1.774619396s" podCreationTimestamp="2026-05-25 11:18:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:13.773888128 +0000 UTC m=+33.785496581" watchObservedRunningTime="2026-05-25 11:18:13.774619396 +0000 UTC m=+33.786227847" May 25 11:18:14.260044 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.260009 2548 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-33.ec2.internal" event="NodeReady" May 25 11:18:14.260506 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.260148 2548 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 25 11:18:14.311782 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.311708 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f9bhg"] May 25 11:18:14.326060 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.326033 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kkhvk"] May 25 11:18:14.326223 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.326206 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.328777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.328756 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 25 11:18:14.329201 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.329181 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 25 11:18:14.329482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.329467 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9k5hq\"" May 25 11:18:14.339582 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.339563 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkhvk"] May 25 11:18:14.339582 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.339585 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f9bhg"] May 25 11:18:14.339733 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.339656 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.341974 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.341953 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 25 11:18:14.342158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.341978 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 25 11:18:14.342304 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.342229 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 25 11:18:14.342405 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.342388 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqrg2\"" May 25 11:18:14.345396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.345375 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:14.345493 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.345417 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:14.345553 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.345533 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:18:14.345553 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.345534 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 25 11:18:14.345653 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.345600 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret podName:79fe35ea-6771-4a9e-ab6f-3790e1c579a1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:46.345582448 +0000 UTC m=+66.357190896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret") pod "global-pull-secret-syncer-8wx4h" (UID: "79fe35ea-6771-4a9e-ab6f-3790e1c579a1") : object "kube-system"/"original-pull-secret" not registered May 25 11:18:14.345653 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.345617 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs podName:8a17bca6-35d7-416a-ab02-2a1c52de2b5a nodeName:}" failed. No retries permitted until 2026-05-25 11:18:46.345608461 +0000 UTC m=+66.357216894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs") pod "network-metrics-daemon-987s2" (UID: "8a17bca6-35d7-416a-ab02-2a1c52de2b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 11:18:14.445770 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445738 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.445918 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445777 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sllx\" (UniqueName: \"kubernetes.io/projected/fac0286b-34ac-45e8-b336-98b16189b98e-kube-api-access-6sllx\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.445918 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445815 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749rt\" (UniqueName: \"kubernetes.io/projected/8bb17520-93f2-4787-a784-45ecb9dca246-kube-api-access-749rt\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.445918 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445889 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bb17520-93f2-4787-a784-45ecb9dca246-config-volume\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.446049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445945 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:14.446049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445979 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.446049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.445994 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb17520-93f2-4787-a784-45ecb9dca246-tmp-dir\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.446213 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.446091 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 11:18:14.446213 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.446128 2548 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 11:18:14.446213 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.446140 2548 projected.go:194] Error preparing data for projected volume kube-api-access-6bmwk for pod openshift-network-diagnostics/network-check-target-czj9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:18:14.446213 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.446192 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk podName:8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:46.446172086 +0000 UTC m=+66.457780524 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmwk" (UniqueName: "kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk") pod "network-check-target-czj9c" (UID: "8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 11:18:14.546336 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546310 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.546488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546343 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb17520-93f2-4787-a784-45ecb9dca246-tmp-dir\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.546488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546383 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.546488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546405 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sllx\" (UniqueName: \"kubernetes.io/projected/fac0286b-34ac-45e8-b336-98b16189b98e-kube-api-access-6sllx\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.546488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546445 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-749rt\" (UniqueName: \"kubernetes.io/projected/8bb17520-93f2-4787-a784-45ecb9dca246-kube-api-access-749rt\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.546488 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.546468 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 11:18:14.546733 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.546491 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:14.546733 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.546531 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls podName:8bb17520-93f2-4787-a784-45ecb9dca246 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.046511212 +0000 UTC m=+35.058119649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls") pod "dns-default-f9bhg" (UID: "8bb17520-93f2-4787-a784-45ecb9dca246") : secret "dns-default-metrics-tls" not found May 25 11:18:14.546733 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:14.546556 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.046539008 +0000 UTC m=+35.058147441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:14.546733 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546473 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bb17520-93f2-4787-a784-45ecb9dca246-config-volume\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.546733 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546721 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb17520-93f2-4787-a784-45ecb9dca246-tmp-dir\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.547009 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.546988 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bb17520-93f2-4787-a784-45ecb9dca246-config-volume\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.558605 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.558579 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749rt\" (UniqueName: \"kubernetes.io/projected/8bb17520-93f2-4787-a784-45ecb9dca246-kube-api-access-749rt\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:14.558697 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.558647 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sllx\" (UniqueName: \"kubernetes.io/projected/fac0286b-34ac-45e8-b336-98b16189b98e-kube-api-access-6sllx\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:14.594760 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.594698 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:14.594863 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.594700 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:14.595073 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.594710 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:14.597276 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597255 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 25 11:18:14.597514 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597494 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcnj6\"" May 25 11:18:14.597591 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597517 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 25 11:18:14.597591 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597540 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 25 11:18:14.597666 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597611 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 25 11:18:14.597756 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:14.597734 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lkpxl\"" May 25 11:18:15.049790 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:15.049754 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:15.049975 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:15.049817 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:15.049975 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:15.049909 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 11:18:15.049975 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:15.049927 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:15.050125 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:15.049986 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.049965559 +0000 UTC m=+36.061574033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:15.050125 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:15.050004 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls podName:8bb17520-93f2-4787-a784-45ecb9dca246 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.049995129 +0000 UTC m=+36.061603563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls") pod "dns-default-f9bhg" (UID: "8bb17520-93f2-4787-a784-45ecb9dca246") : secret "dns-default-metrics-tls" not found May 25 11:18:16.059518 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:16.059479 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:16.059954 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:16.059546 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:16.059954 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:16.059653 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 11:18:16.059954 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:16.059670 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:16.059954 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:16.059732 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls podName:8bb17520-93f2-4787-a784-45ecb9dca246 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.059711026 +0000 UTC m=+38.071319462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls") pod "dns-default-f9bhg" (UID: "8bb17520-93f2-4787-a784-45ecb9dca246") : secret "dns-default-metrics-tls" not found May 25 11:18:16.059954 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:16.059756 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.059742669 +0000 UTC m=+38.071351107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:17.770122 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:17.770065 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="e4232c95f64aebbca1dd2830521047a9496b9bf54df43aaf5808ea8aa283846a" exitCode=0 May 25 11:18:17.770477 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:17.770144 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"e4232c95f64aebbca1dd2830521047a9496b9bf54df43aaf5808ea8aa283846a"} May 25 11:18:18.072477 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:18.072384 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:18.072477 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:18.072439 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:18.072664 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:18.072524 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:18.072664 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:18.072528 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 11:18:18.072664 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:18.072576 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:22.072561719 +0000 UTC m=+42.084170152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:18.072664 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:18.072589 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls podName:8bb17520-93f2-4787-a784-45ecb9dca246 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:22.072583613 +0000 UTC m=+42.084192047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls") pod "dns-default-f9bhg" (UID: "8bb17520-93f2-4787-a784-45ecb9dca246") : secret "dns-default-metrics-tls" not found May 25 11:18:18.774082 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:18.774049 2548 generic.go:358] "Generic (PLEG): container finished" podID="82daf91e-0f98-4b15-ba16-d1732291b935" containerID="0ac886d083034f12bbaad3e64ad881d5cf71da292668379b51f58e706cd3f2da" exitCode=0 May 25 11:18:18.774516 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:18.774092 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerDied","Data":"0ac886d083034f12bbaad3e64ad881d5cf71da292668379b51f58e706cd3f2da"} May 25 11:18:19.778531 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:19.778492 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-58hq2" event={"ID":"82daf91e-0f98-4b15-ba16-d1732291b935","Type":"ContainerStarted","Data":"928de32557b94660c4b7db37207c629af36b155ad14899779dcdf3e275cae49f"} May 25 11:18:19.813977 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:19.813931 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-58hq2" podStartSLOduration=5.729016752 podStartE2EDuration="39.813920155s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:17:43.214112342 +0000 UTC m=+3.225720780" lastFinishedPulling="2026-05-25 11:18:17.299015744 +0000 UTC m=+37.310624183" observedRunningTime="2026-05-25 11:18:19.811715312 +0000 UTC m=+39.823323768" watchObservedRunningTime="2026-05-25 11:18:19.813920155 +0000 UTC m=+39.825528610" May 25 11:18:20.678856 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:20.678829 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jf4zc_06d68c20-0bdc-4cc2-8357-885bba30014f/dns-node-resolver/0.log" May 25 11:18:21.295859 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.295824 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hmlf2"] May 25 11:18:21.301415 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.301398 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.305957 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.305936 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" May 25 11:18:21.306132 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.306092 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" May 25 11:18:21.307009 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.306993 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" May 25 11:18:21.307393 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.307375 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" May 25 11:18:21.307393 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.307387 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dvsbf\"" May 25 11:18:21.309010 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.308989 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hmlf2"] May 25 11:18:21.396524 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.396500 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-key\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.396653 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.396553 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-cabundle\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.396653 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.396619 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pm8\" (UniqueName: \"kubernetes.io/projected/3340bf57-0cb1-4e53-ba84-feef6f7b1560-kube-api-access-29pm8\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.497111 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.497085 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-key\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.497194 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.497146 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-cabundle\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.497291 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.497274 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29pm8\" (UniqueName: \"kubernetes.io/projected/3340bf57-0cb1-4e53-ba84-feef6f7b1560-kube-api-access-29pm8\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.497839 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.497820 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-cabundle\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.500571 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.500542 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3340bf57-0cb1-4e53-ba84-feef6f7b1560-signing-key\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.511463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.511439 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pm8\" (UniqueName: \"kubernetes.io/projected/3340bf57-0cb1-4e53-ba84-feef6f7b1560-kube-api-access-29pm8\") pod \"service-ca-67d667c7d-hmlf2\" (UID: \"3340bf57-0cb1-4e53-ba84-feef6f7b1560\") " pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.611459 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.611438 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" May 25 11:18:21.678031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.677997 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwh8j_2f9b0c54-3b7b-44e1-9614-0890e21e2157/node-ca/0.log" May 25 11:18:21.746473 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.746444 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hmlf2"] May 25 11:18:21.749837 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:21.749802 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3340bf57_0cb1_4e53_ba84_feef6f7b1560.slice/crio-5936b394719aa43d225ef86ba4b85f963d6066a0e4ea6c8dfd9b0d363daa4730 WatchSource:0}: Error finding container 5936b394719aa43d225ef86ba4b85f963d6066a0e4ea6c8dfd9b0d363daa4730: Status 404 returned error can't find the container with id 5936b394719aa43d225ef86ba4b85f963d6066a0e4ea6c8dfd9b0d363daa4730 May 25 11:18:21.783369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:21.783343 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" event={"ID":"3340bf57-0cb1-4e53-ba84-feef6f7b1560","Type":"ContainerStarted","Data":"5936b394719aa43d225ef86ba4b85f963d6066a0e4ea6c8dfd9b0d363daa4730"} May 25 11:18:22.102255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:22.102179 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:22.102255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:22.102225 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:22.102413 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:22.102339 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 11:18:22.102413 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:22.102351 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:22.102413 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:22.102412 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls podName:8bb17520-93f2-4787-a784-45ecb9dca246 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:30.102384418 +0000 UTC m=+50.113992850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls") pod "dns-default-f9bhg" (UID: "8bb17520-93f2-4787-a784-45ecb9dca246") : secret "dns-default-metrics-tls" not found May 25 11:18:22.102519 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:22.102428 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:30.102421874 +0000 UTC m=+50.114030308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:23.789376 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:23.789342 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" event={"ID":"3340bf57-0cb1-4e53-ba84-feef6f7b1560","Type":"ContainerStarted","Data":"8927bc17b3333440936c8f5dabed40eee62574f85e9cd886cac1a24dfe7e191d"} May 25 11:18:23.821627 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:23.821582 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-hmlf2" podStartSLOduration=0.890754816 podStartE2EDuration="2.821568682s" podCreationTimestamp="2026-05-25 11:18:21 +0000 UTC" firstStartedPulling="2026-05-25 11:18:21.751655719 +0000 UTC m=+41.763264153" lastFinishedPulling="2026-05-25 11:18:23.682469581 +0000 UTC m=+43.694078019" observedRunningTime="2026-05-25 11:18:23.820785684 +0000 UTC m=+43.832394150" watchObservedRunningTime="2026-05-25 11:18:23.821568682 +0000 UTC m=+43.833177137" May 25 11:18:30.161965 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.161922 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:30.162379 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.161974 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:30.162379 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:30.162074 2548 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 11:18:30.162379 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:18:30.162167 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert podName:fac0286b-34ac-45e8-b336-98b16189b98e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:46.162145854 +0000 UTC m=+66.173754292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert") pod "ingress-canary-kkhvk" (UID: "fac0286b-34ac-45e8-b336-98b16189b98e") : secret "canary-serving-cert" not found May 25 11:18:30.164513 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.164487 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bb17520-93f2-4787-a784-45ecb9dca246-metrics-tls\") pod \"dns-default-f9bhg\" (UID: \"8bb17520-93f2-4787-a784-45ecb9dca246\") " pod="openshift-dns/dns-default-f9bhg" May 25 11:18:30.238265 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.238221 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f9bhg" May 25 11:18:30.364240 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.364212 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f9bhg"] May 25 11:18:30.372972 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:30.372939 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb17520_93f2_4787_a784_45ecb9dca246.slice/crio-2d09521870b55570c3590a0025f079519321027c54e274a3727af02dbf8b1536 WatchSource:0}: Error finding container 2d09521870b55570c3590a0025f079519321027c54e274a3727af02dbf8b1536: Status 404 returned error can't find the container with id 2d09521870b55570c3590a0025f079519321027c54e274a3727af02dbf8b1536 May 25 11:18:30.805396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:30.805354 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9bhg" event={"ID":"8bb17520-93f2-4787-a784-45ecb9dca246","Type":"ContainerStarted","Data":"2d09521870b55570c3590a0025f079519321027c54e274a3727af02dbf8b1536"} May 25 11:18:32.810647 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:32.810611 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9bhg" event={"ID":"8bb17520-93f2-4787-a784-45ecb9dca246","Type":"ContainerStarted","Data":"e1fde3ac1e2db4e48dac6a613d8f9286f9ea02e9fd4520b0be73095ba8ad2752"} May 25 11:18:32.810647 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:32.810647 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9bhg" event={"ID":"8bb17520-93f2-4787-a784-45ecb9dca246","Type":"ContainerStarted","Data":"d42de9c9144740cf31870c0a7972effb24528261f1a8b1fbe22e041c34a0b0d8"} May 25 11:18:32.811166 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:32.810796 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f9bhg" May 25 11:18:32.835758 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:32.835708 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-f9bhg" podStartSLOduration=17.179717418 podStartE2EDuration="18.835695676s" podCreationTimestamp="2026-05-25 11:18:14 +0000 UTC" firstStartedPulling="2026-05-25 11:18:30.374903128 +0000 UTC m=+50.386511561" lastFinishedPulling="2026-05-25 11:18:32.030881373 +0000 UTC m=+52.042489819" observedRunningTime="2026-05-25 11:18:32.835516857 +0000 UTC m=+52.847125328" watchObservedRunningTime="2026-05-25 11:18:32.835695676 +0000 UTC m=+52.847304131" May 25 11:18:40.761543 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:40.761514 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmtvc" May 25 11:18:42.815251 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:42.815212 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f9bhg" May 25 11:18:45.028406 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.028368 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6zwgb"] May 25 11:18:45.030801 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.030786 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.034043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.034014 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 25 11:18:45.034043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.034023 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 25 11:18:45.034319 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.034304 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 25 11:18:45.034319 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.034312 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9btq\"" May 25 11:18:45.034442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.034340 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 25 11:18:45.049363 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.049337 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zwgb"] May 25 11:18:45.060657 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.060636 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/859924b5-0863-4e63-969c-fb4bb29d3863-crio-socket\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.060761 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.060666 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/859924b5-0863-4e63-969c-fb4bb29d3863-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.060761 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.060695 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cfz\" (UniqueName: \"kubernetes.io/projected/859924b5-0863-4e63-969c-fb4bb29d3863-kube-api-access-j5cfz\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.060761 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.060754 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/859924b5-0863-4e63-969c-fb4bb29d3863-data-volume\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.060885 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.060791 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/859924b5-0863-4e63-969c-fb4bb29d3863-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.128492 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.128469 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7"] May 25 11:18:45.130237 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.130222 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.134915 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.134893 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk"] May 25 11:18:45.136894 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.136875 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.142078 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.142059 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nscl4\"" May 25 11:18:45.157213 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157194 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" May 25 11:18:45.157499 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157473 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" May 25 11:18:45.157575 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157496 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" May 25 11:18:45.157575 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157534 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" May 25 11:18:45.157575 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157549 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" May 25 11:18:45.157824 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157777 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" May 25 11:18:45.157921 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.157910 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" May 25 11:18:45.158204 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.158188 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" May 25 11:18:45.158251 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.158190 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" May 25 11:18:45.161377 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161361 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cfz\" (UniqueName: \"kubernetes.io/projected/859924b5-0863-4e63-969c-fb4bb29d3863-kube-api-access-j5cfz\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161387 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161412 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/859924b5-0863-4e63-969c-fb4bb29d3863-data-volume\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161550 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161530 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23080302-2bff-48d8-a38a-df55f540ae20-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.161607 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161578 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161616 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/859924b5-0863-4e63-969c-fb4bb29d3863-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161716 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161654 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23080302-2bff-48d8-a38a-df55f540ae20-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.161716 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161686 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/859924b5-0863-4e63-969c-fb4bb29d3863-data-volume\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161716 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161691 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161867 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161719 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqlw5\" (UniqueName: \"kubernetes.io/projected/3a714abd-a6bd-404a-bbff-f6f1b7800e26-kube-api-access-bqlw5\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161867 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161755 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161867 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161861 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/859924b5-0863-4e63-969c-fb4bb29d3863-crio-socket\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161990 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161881 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/859924b5-0863-4e63-969c-fb4bb29d3863-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.161990 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161899 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.161990 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.161982 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/859924b5-0863-4e63-969c-fb4bb29d3863-crio-socket\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.162325 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.162310 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/859924b5-0863-4e63-969c-fb4bb29d3863-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.163805 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.163789 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/859924b5-0863-4e63-969c-fb4bb29d3863-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.170448 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.170426 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w"] May 25 11:18:45.172070 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.172045 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk"] May 25 11:18:45.172180 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.172166 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.175547 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.175520 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7"] May 25 11:18:45.175869 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.175851 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pn8j9\"" May 25 11:18:45.175869 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.175860 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" May 25 11:18:45.188609 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.188592 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cfz\" (UniqueName: \"kubernetes.io/projected/859924b5-0863-4e63-969c-fb4bb29d3863-kube-api-access-j5cfz\") pod \"insights-runtime-extractor-6zwgb\" (UID: \"859924b5-0863-4e63-969c-fb4bb29d3863\") " pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.205360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.205330 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bc79c8b59-mfwq2"] May 25 11:18:45.207288 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.207273 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w"] May 25 11:18:45.207367 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.207356 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.210169 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.210151 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 25 11:18:45.210268 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.210172 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh8tr\"" May 25 11:18:45.210268 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.210181 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 25 11:18:45.210268 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.210194 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 25 11:18:45.220068 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.220052 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 25 11:18:45.255854 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.255823 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc79c8b59-mfwq2"] May 25 11:18:45.262267 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262249 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7cc\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-kube-api-access-gk7cc\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262276 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-installation-pull-secrets\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262301 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/175c80c9-e375-497b-9e9e-fa9685cba2e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.262349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262329 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262349 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23080302-2bff-48d8-a38a-df55f540ae20-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.262481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262373 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262399 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqlw5\" (UniqueName: \"kubernetes.io/projected/3a714abd-a6bd-404a-bbff-f6f1b7800e26-kube-api-access-bqlw5\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262442 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262479 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-bound-sa-token\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262506 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-image-registry-private-configuration\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262564 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262579 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-certificates\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262615 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25kl\" (UniqueName: \"kubernetes.io/projected/175c80c9-e375-497b-9e9e-fa9685cba2e6-kube-api-access-s25kl\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.262887 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262752 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.262887 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262792 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23080302-2bff-48d8-a38a-df55f540ae20-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.262887 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262836 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9baa0b4d-a60a-468d-a4dc-d8f52596706e-ca-trust-extracted\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.262887 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262870 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-trusted-ca\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.263116 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.262899 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-tls\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.263232 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.263212 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.263296 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.263212 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23080302-2bff-48d8-a38a-df55f540ae20-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.265332 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.265307 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-ca\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.265418 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.265390 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.265611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.265588 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23080302-2bff-48d8-a38a-df55f540ae20-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-vx9c7\" (UID: \"23080302-2bff-48d8-a38a-df55f540ae20\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.265757 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.265741 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.265866 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.265849 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a714abd-a6bd-404a-bbff-f6f1b7800e26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.318496 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.318430 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqlw5\" (UniqueName: \"kubernetes.io/projected/3a714abd-a6bd-404a-bbff-f6f1b7800e26-kube-api-access-bqlw5\") pod \"cluster-proxy-proxy-agent-7c86c4cff6-mv7xk\" (UID: \"3a714abd-a6bd-404a-bbff-f6f1b7800e26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.339661 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.339640 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zwgb" May 25 11:18:45.364228 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364198 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-bound-sa-token\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364345 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364245 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-image-registry-private-configuration\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364471 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364451 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-certificates\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364526 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364501 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s25kl\" (UniqueName: \"kubernetes.io/projected/175c80c9-e375-497b-9e9e-fa9685cba2e6-kube-api-access-s25kl\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.364635 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364541 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9baa0b4d-a60a-468d-a4dc-d8f52596706e-ca-trust-extracted\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364635 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364565 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-trusted-ca\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364635 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364589 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-tls\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364633 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7cc\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-kube-api-access-gk7cc\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364666 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-installation-pull-secrets\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.364791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.364692 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/175c80c9-e375-497b-9e9e-fa9685cba2e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.365343 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.365289 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9baa0b4d-a60a-468d-a4dc-d8f52596706e-ca-trust-extracted\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.365446 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.365340 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-certificates\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.365503 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.365484 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9baa0b4d-a60a-468d-a4dc-d8f52596706e-trusted-ca\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.367261 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.367238 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-image-registry-private-configuration\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.367361 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.367329 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9baa0b4d-a60a-468d-a4dc-d8f52596706e-installation-pull-secrets\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.367837 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.367806 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/175c80c9-e375-497b-9e9e-fa9685cba2e6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.367922 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.367876 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-registry-tls\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.393821 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.393758 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25kl\" (UniqueName: \"kubernetes.io/projected/175c80c9-e375-497b-9e9e-fa9685cba2e6-kube-api-access-s25kl\") pod \"managed-serviceaccount-addon-agent-5f5b98d44-lc54w\" (UID: \"175c80c9-e375-497b-9e9e-fa9685cba2e6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.393955 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.393902 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7cc\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-kube-api-access-gk7cc\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.395062 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.395037 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baa0b4d-a60a-468d-a4dc-d8f52596706e-bound-sa-token\") pod \"image-registry-bc79c8b59-mfwq2\" (UID: \"9baa0b4d-a60a-468d-a4dc-d8f52596706e\") " pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.441365 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.441337 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" May 25 11:18:45.468823 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.468792 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" May 25 11:18:45.482553 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.482528 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" May 25 11:18:45.494914 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.494340 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zwgb"] May 25 11:18:45.533805 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.533778 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.594268 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.594233 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7"] May 25 11:18:45.597687 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:45.597653 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23080302_2bff_48d8_a38a_df55f540ae20.slice/crio-acf42446bd63474a16d0d7562553599bbeec4da129a9efe4b3b54657be9fbda3 WatchSource:0}: Error finding container acf42446bd63474a16d0d7562553599bbeec4da129a9efe4b3b54657be9fbda3: Status 404 returned error can't find the container with id acf42446bd63474a16d0d7562553599bbeec4da129a9efe4b3b54657be9fbda3 May 25 11:18:45.649222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.649137 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk"] May 25 11:18:45.652879 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:45.652845 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a714abd_a6bd_404a_bbff_f6f1b7800e26.slice/crio-a6c19e09a9c069428de77149165f933e204861addc75ef4addbd8cbd27303067 WatchSource:0}: Error finding container a6c19e09a9c069428de77149165f933e204861addc75ef4addbd8cbd27303067: Status 404 returned error can't find the container with id a6c19e09a9c069428de77149165f933e204861addc75ef4addbd8cbd27303067 May 25 11:18:45.668630 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.668605 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w"] May 25 11:18:45.671917 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:45.671896 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod175c80c9_e375_497b_9e9e_fa9685cba2e6.slice/crio-554e936599fa2fe236fa0bb18b69fa1ab42cee9807515966f116e3fac454bb13 WatchSource:0}: Error finding container 554e936599fa2fe236fa0bb18b69fa1ab42cee9807515966f116e3fac454bb13: Status 404 returned error can't find the container with id 554e936599fa2fe236fa0bb18b69fa1ab42cee9807515966f116e3fac454bb13 May 25 11:18:45.705642 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.705618 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc79c8b59-mfwq2"] May 25 11:18:45.709818 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:45.709789 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9baa0b4d_a60a_468d_a4dc_d8f52596706e.slice/crio-09a79c710c63a56a2e7bf7a592c8f5b99af76e9d8b62bedfb802e47ee0e07681 WatchSource:0}: Error finding container 09a79c710c63a56a2e7bf7a592c8f5b99af76e9d8b62bedfb802e47ee0e07681: Status 404 returned error can't find the container with id 09a79c710c63a56a2e7bf7a592c8f5b99af76e9d8b62bedfb802e47ee0e07681 May 25 11:18:45.839284 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.839219 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" event={"ID":"9baa0b4d-a60a-468d-a4dc-d8f52596706e","Type":"ContainerStarted","Data":"f47dd382a55f2f8ce39c3a1f0dac9c51d66128bde43737b3f68457bf580904c2"} May 25 11:18:45.839284 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.839254 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" event={"ID":"9baa0b4d-a60a-468d-a4dc-d8f52596706e","Type":"ContainerStarted","Data":"09a79c710c63a56a2e7bf7a592c8f5b99af76e9d8b62bedfb802e47ee0e07681"} May 25 11:18:45.839476 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.839354 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:18:45.840372 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.840346 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" event={"ID":"3a714abd-a6bd-404a-bbff-f6f1b7800e26","Type":"ContainerStarted","Data":"a6c19e09a9c069428de77149165f933e204861addc75ef4addbd8cbd27303067"} May 25 11:18:45.841387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.841356 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" event={"ID":"23080302-2bff-48d8-a38a-df55f540ae20","Type":"ContainerStarted","Data":"acf42446bd63474a16d0d7562553599bbeec4da129a9efe4b3b54657be9fbda3"} May 25 11:18:45.842737 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.842710 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zwgb" event={"ID":"859924b5-0863-4e63-969c-fb4bb29d3863","Type":"ContainerStarted","Data":"f0930f57978bfde75f7048564f23ec6df22ee781f3f83ad1f728dd672e2ceb07"} May 25 11:18:45.842869 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.842744 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zwgb" event={"ID":"859924b5-0863-4e63-969c-fb4bb29d3863","Type":"ContainerStarted","Data":"1d8e47f8dcbe42ab0457f602b50db86d1ee6fe1c2d610d2299eab6c26b551b73"} May 25 11:18:45.843615 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:45.843593 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" event={"ID":"175c80c9-e375-497b-9e9e-fa9685cba2e6","Type":"ContainerStarted","Data":"554e936599fa2fe236fa0bb18b69fa1ab42cee9807515966f116e3fac454bb13"} May 25 11:18:46.174158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.174094 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:46.177548 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.177509 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac0286b-34ac-45e8-b336-98b16189b98e-cert\") pod \"ingress-canary-kkhvk\" (UID: \"fac0286b-34ac-45e8-b336-98b16189b98e\") " pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:46.375897 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.375863 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:46.376034 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.375948 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:46.379026 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.379001 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 25 11:18:46.379499 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.379293 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 25 11:18:46.390430 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.390368 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe35ea-6771-4a9e-ab6f-3790e1c579a1-original-pull-secret\") pod \"global-pull-secret-syncer-8wx4h\" (UID: \"79fe35ea-6771-4a9e-ab6f-3790e1c579a1\") " pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:46.392810 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.392786 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a17bca6-35d7-416a-ab02-2a1c52de2b5a-metrics-certs\") pod \"network-metrics-daemon-987s2\" (UID: \"8a17bca6-35d7-416a-ab02-2a1c52de2b5a\") " pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:46.413301 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.412962 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wx4h" May 25 11:18:46.421026 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.420845 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lkpxl\"" May 25 11:18:46.429396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.429058 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-987s2" May 25 11:18:46.459400 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.458346 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqrg2\"" May 25 11:18:46.465987 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.465604 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkhvk" May 25 11:18:46.477402 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.476870 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:46.479548 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.479349 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 25 11:18:46.494672 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.493790 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 25 11:18:46.527485 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.527360 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmwk\" (UniqueName: \"kubernetes.io/projected/8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1-kube-api-access-6bmwk\") pod \"network-check-target-czj9c\" (UID: \"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1\") " pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:46.617031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.616230 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" podStartSLOduration=1.6162082359999999 podStartE2EDuration="1.616208236s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:45.878349839 +0000 UTC m=+65.889958294" watchObservedRunningTime="2026-05-25 11:18:46.616208236 +0000 UTC m=+66.627816689" May 25 11:18:46.617031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.616987 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8wx4h"] May 25 11:18:46.629397 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:46.629367 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe35ea_6771_4a9e_ab6f_3790e1c579a1.slice/crio-1d763e9a33770e3ce5ca733878701237b42d262527ffafa108babe6a78ad0aef WatchSource:0}: Error finding container 1d763e9a33770e3ce5ca733878701237b42d262527ffafa108babe6a78ad0aef: Status 404 returned error can't find the container with id 1d763e9a33770e3ce5ca733878701237b42d262527ffafa108babe6a78ad0aef May 25 11:18:46.639789 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.639764 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-987s2"] May 25 11:18:46.649197 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:46.649166 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a17bca6_35d7_416a_ab02_2a1c52de2b5a.slice/crio-16ce885c0f370da09948fb9451750b62bc011af5ca605548eef1ec3f1bb3644d WatchSource:0}: Error finding container 16ce885c0f370da09948fb9451750b62bc011af5ca605548eef1ec3f1bb3644d: Status 404 returned error can't find the container with id 16ce885c0f370da09948fb9451750b62bc011af5ca605548eef1ec3f1bb3644d May 25 11:18:46.685465 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.685372 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkhvk"] May 25 11:18:46.708807 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.708597 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcnj6\"" May 25 11:18:46.717134 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.716754 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:46.859581 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.859541 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zwgb" event={"ID":"859924b5-0863-4e63-969c-fb4bb29d3863","Type":"ContainerStarted","Data":"9880d861bfc8540fa0bbb0daae0d310fd0b4f009849af17b6b29700d4f35daa6"} May 25 11:18:46.861878 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.861847 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-987s2" event={"ID":"8a17bca6-35d7-416a-ab02-2a1c52de2b5a","Type":"ContainerStarted","Data":"16ce885c0f370da09948fb9451750b62bc011af5ca605548eef1ec3f1bb3644d"} May 25 11:18:46.864209 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:46.864029 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8wx4h" event={"ID":"79fe35ea-6771-4a9e-ab6f-3790e1c579a1","Type":"ContainerStarted","Data":"1d763e9a33770e3ce5ca733878701237b42d262527ffafa108babe6a78ad0aef"} May 25 11:18:47.329014 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:47.328973 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac0286b_34ac_45e8_b336_98b16189b98e.slice/crio-d7ce01789174b647c1169c8a519290bcf2aa0a9831dd42687ee52042a171ebcd WatchSource:0}: Error finding container d7ce01789174b647c1169c8a519290bcf2aa0a9831dd42687ee52042a171ebcd: Status 404 returned error can't find the container with id d7ce01789174b647c1169c8a519290bcf2aa0a9831dd42687ee52042a171ebcd May 25 11:18:47.524037 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:47.523994 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-czj9c"] May 25 11:18:47.870407 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:47.870366 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" event={"ID":"23080302-2bff-48d8-a38a-df55f540ae20","Type":"ContainerStarted","Data":"061548fb9c45742700c5083a4983f919a4ed78c760a302494fd8d30929873e85"} May 25 11:18:47.872760 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:47.872452 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkhvk" event={"ID":"fac0286b-34ac-45e8-b336-98b16189b98e","Type":"ContainerStarted","Data":"d7ce01789174b647c1169c8a519290bcf2aa0a9831dd42687ee52042a171ebcd"} May 25 11:18:47.916456 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:47.916027 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-vx9c7" podStartSLOduration=1.171493062 podStartE2EDuration="2.916007432s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.603475191 +0000 UTC m=+65.615083629" lastFinishedPulling="2026-05-25 11:18:47.34798956 +0000 UTC m=+67.359597999" observedRunningTime="2026-05-25 11:18:47.914611407 +0000 UTC m=+67.926219863" watchObservedRunningTime="2026-05-25 11:18:47.916007432 +0000 UTC m=+67.927615891" May 25 11:18:48.873165 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:18:48.873077 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d60e59c_fecb_4f0f_aa3d_f29793dbbfc1.slice/crio-d86bdd77d4f900f32afd010232c5df1a1bf3db4b95c136ee5b6d6d73ac537de7 WatchSource:0}: Error finding container d86bdd77d4f900f32afd010232c5df1a1bf3db4b95c136ee5b6d6d73ac537de7: Status 404 returned error can't find the container with id d86bdd77d4f900f32afd010232c5df1a1bf3db4b95c136ee5b6d6d73ac537de7 May 25 11:18:49.879888 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:49.879844 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-czj9c" event={"ID":"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1","Type":"ContainerStarted","Data":"d86bdd77d4f900f32afd010232c5df1a1bf3db4b95c136ee5b6d6d73ac537de7"} May 25 11:18:54.896627 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.896587 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" event={"ID":"175c80c9-e375-497b-9e9e-fa9685cba2e6","Type":"ContainerStarted","Data":"e966e65211e75ab3b9d94c83d77f4bf08b13e812cf784e7a4e7312d66e81b04a"} May 25 11:18:54.898334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.898302 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" event={"ID":"3a714abd-a6bd-404a-bbff-f6f1b7800e26","Type":"ContainerStarted","Data":"5208f43f95265ec46a0d6282acc559f7ab116e02d9f4eb9a9905c5641d130929"} May 25 11:18:54.900466 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.900440 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zwgb" event={"ID":"859924b5-0863-4e63-969c-fb4bb29d3863","Type":"ContainerStarted","Data":"f13a269a6c8540a7d333739a157ab33f57c0f5827a52da654a51c2c8272a79e5"} May 25 11:18:54.902085 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.902050 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkhvk" event={"ID":"fac0286b-34ac-45e8-b336-98b16189b98e","Type":"ContainerStarted","Data":"dced772cd77af0849875962a2c79f3d375e97db92b602d457a6805835b5fc2f2"} May 25 11:18:54.903720 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.903690 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-987s2" event={"ID":"8a17bca6-35d7-416a-ab02-2a1c52de2b5a","Type":"ContainerStarted","Data":"194dd6778a3f431be9d1865fb375ede86e19ad7f6a026703c6834c86807babd0"} May 25 11:18:54.903842 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.903722 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-987s2" event={"ID":"8a17bca6-35d7-416a-ab02-2a1c52de2b5a","Type":"ContainerStarted","Data":"41790b386c4b53b3657150ab3ea3b4148ce7a86c63f3f8394f109637c9f7432e"} May 25 11:18:54.905367 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.905346 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8wx4h" event={"ID":"79fe35ea-6771-4a9e-ab6f-3790e1c579a1","Type":"ContainerStarted","Data":"b32430157e5de31c9dd4fe0742e335ad334265f96b40621f43785eac1e2deb76"} May 25 11:18:54.906949 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.906923 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-czj9c" event={"ID":"8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1","Type":"ContainerStarted","Data":"c3b1c2a57930571b0d9777519eec7b0d3194f73751ce9f5b6109ddf7ef22950a"} May 25 11:18:54.907158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.907131 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:18:54.932267 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.931016 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f5b98d44-lc54w" podStartSLOduration=2.519150122 podStartE2EDuration="9.931002573s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.673642059 +0000 UTC m=+65.685250492" lastFinishedPulling="2026-05-25 11:18:53.085494492 +0000 UTC m=+73.097102943" observedRunningTime="2026-05-25 11:18:54.911506141 +0000 UTC m=+74.923114596" watchObservedRunningTime="2026-05-25 11:18:54.931002573 +0000 UTC m=+74.942611029" May 25 11:18:54.932267 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.931258 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-987s2" podStartSLOduration=67.528895047 podStartE2EDuration="1m14.93124923s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:18:46.65223457 +0000 UTC m=+66.663843007" lastFinishedPulling="2026-05-25 11:18:54.054588756 +0000 UTC m=+74.066197190" observedRunningTime="2026-05-25 11:18:54.930390891 +0000 UTC m=+74.941999346" watchObservedRunningTime="2026-05-25 11:18:54.93124923 +0000 UTC m=+74.942857686" May 25 11:18:54.948137 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.947410 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kkhvk" podStartSLOduration=34.227128752 podStartE2EDuration="40.947395448s" podCreationTimestamp="2026-05-25 11:18:14 +0000 UTC" firstStartedPulling="2026-05-25 11:18:47.34344396 +0000 UTC m=+67.355052398" lastFinishedPulling="2026-05-25 11:18:54.063710647 +0000 UTC m=+74.075319094" observedRunningTime="2026-05-25 11:18:54.947236216 +0000 UTC m=+74.958844671" watchObservedRunningTime="2026-05-25 11:18:54.947395448 +0000 UTC m=+74.959003906" May 25 11:18:54.964754 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.964703 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8wx4h" podStartSLOduration=66.541062092 podStartE2EDuration="1m13.964687109s" podCreationTimestamp="2026-05-25 11:17:41 +0000 UTC" firstStartedPulling="2026-05-25 11:18:46.634492461 +0000 UTC m=+66.646100897" lastFinishedPulling="2026-05-25 11:18:54.058117474 +0000 UTC m=+74.069725914" observedRunningTime="2026-05-25 11:18:54.963725945 +0000 UTC m=+74.975334400" watchObservedRunningTime="2026-05-25 11:18:54.964687109 +0000 UTC m=+74.976295565" May 25 11:18:54.983183 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.983086 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6zwgb" podStartSLOduration=1.530747018 podStartE2EDuration="9.983072188s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.60218092 +0000 UTC m=+65.613789353" lastFinishedPulling="2026-05-25 11:18:54.054506072 +0000 UTC m=+74.066114523" observedRunningTime="2026-05-25 11:18:54.982555768 +0000 UTC m=+74.994164224" watchObservedRunningTime="2026-05-25 11:18:54.983072188 +0000 UTC m=+74.994680645" May 25 11:18:54.999379 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:54.998817 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-czj9c" podStartSLOduration=69.80453715 podStartE2EDuration="1m14.998804093s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:18:48.876297144 +0000 UTC m=+68.887905576" lastFinishedPulling="2026-05-25 11:18:54.070564078 +0000 UTC m=+74.082172519" observedRunningTime="2026-05-25 11:18:54.996741388 +0000 UTC m=+75.008349846" watchObservedRunningTime="2026-05-25 11:18:54.998804093 +0000 UTC m=+75.010412549" May 25 11:18:56.913586 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:56.913517 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" event={"ID":"3a714abd-a6bd-404a-bbff-f6f1b7800e26","Type":"ContainerStarted","Data":"e62a928a3484473fe13f0c71bcf68623e6f0c5422c5fa608bac5d1ee2415afc4"} May 25 11:18:56.913586 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:56.913550 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" event={"ID":"3a714abd-a6bd-404a-bbff-f6f1b7800e26","Type":"ContainerStarted","Data":"f8c594f25b73bf07b2e8005ed0c5686040005ee8f9461ae27a17f897f94b6f3b"} May 25 11:18:56.933766 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:18:56.933721 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86c4cff6-mv7xk" podStartSLOduration=0.924471334 podStartE2EDuration="11.933706874s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.655565323 +0000 UTC m=+65.667173756" lastFinishedPulling="2026-05-25 11:18:56.664800857 +0000 UTC m=+76.676409296" observedRunningTime="2026-05-25 11:18:56.932159962 +0000 UTC m=+76.943768417" watchObservedRunningTime="2026-05-25 11:18:56.933706874 +0000 UTC m=+76.945315329" May 25 11:19:06.869182 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:06.869152 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bc79c8b59-mfwq2" May 25 11:19:07.190122 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.190025 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-99c7k"] May 25 11:19:07.195787 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.195767 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.199344 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.199319 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 25 11:19:07.199463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.199361 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 25 11:19:07.199463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.199363 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 25 11:19:07.199463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.199325 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 25 11:19:07.199633 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.199369 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 25 11:19:07.200767 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.200744 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 25 11:19:07.200867 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.200770 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qlxnt\"" May 25 11:19:07.231822 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.231656 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-sys\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.231986 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.231968 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-tls\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232090 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232076 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-metrics-client-ca\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232212 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232197 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdhq\" (UniqueName: \"kubernetes.io/projected/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-kube-api-access-skdhq\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232327 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232313 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-root\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232437 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232424 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-wtmp\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232555 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232540 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232653 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232641 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-textfile\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.232794 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.232780 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-accelerators-collector-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333543 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333516 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-accelerators-collector-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333549 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-sys\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333565 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-tls\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333580 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-metrics-client-ca\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333655 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333605 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-skdhq\" (UniqueName: \"kubernetes.io/projected/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-kube-api-access-skdhq\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333650 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-sys\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333714 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-root\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333765 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-wtmp\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333791 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333802 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-root\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.333849 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333816 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-textfile\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.334158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.333960 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-wtmp\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.334158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.334135 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-textfile\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.334255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.334227 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-accelerators-collector-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.334309 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.334258 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-metrics-client-ca\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.336035 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.336015 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-tls\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.336198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.336178 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.342023 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.341996 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdhq\" (UniqueName: \"kubernetes.io/projected/e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92-kube-api-access-skdhq\") pod \"node-exporter-99c7k\" (UID: \"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92\") " pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.506221 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.506165 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99c7k" May 25 11:19:07.514083 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:19:07.514057 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c8aa98_ec36_47fa_8d8f_f14b8c5a6a92.slice/crio-196b7ef0390bf49a220e9c2edfe46bab24d20f77f8f0938a0a459035c4163ca6 WatchSource:0}: Error finding container 196b7ef0390bf49a220e9c2edfe46bab24d20f77f8f0938a0a459035c4163ca6: Status 404 returned error can't find the container with id 196b7ef0390bf49a220e9c2edfe46bab24d20f77f8f0938a0a459035c4163ca6 May 25 11:19:07.941359 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:07.941324 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99c7k" event={"ID":"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92","Type":"ContainerStarted","Data":"196b7ef0390bf49a220e9c2edfe46bab24d20f77f8f0938a0a459035c4163ca6"} May 25 11:19:08.945400 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:08.945367 2548 generic.go:358] "Generic (PLEG): container finished" podID="e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92" containerID="e256983190ab896c47cfeff3d0e6329cf200dbab2307b742014b37a1dadb41a6" exitCode=0 May 25 11:19:08.945726 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:08.945450 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99c7k" event={"ID":"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92","Type":"ContainerDied","Data":"e256983190ab896c47cfeff3d0e6329cf200dbab2307b742014b37a1dadb41a6"} May 25 11:19:09.950201 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:09.950166 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99c7k" event={"ID":"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92","Type":"ContainerStarted","Data":"8443394c79c531e5a38bbc7829ac677db964b99bc67bcf41bd181ebbe2971362"} May 25 11:19:09.950201 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:09.950203 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99c7k" event={"ID":"e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92","Type":"ContainerStarted","Data":"9f5e47174af0b4160336597fc621b60360534034dcc3a761bb4ccb3f84bb2030"} May 25 11:19:09.971387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:09.971338 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-99c7k" podStartSLOduration=1.966755713 podStartE2EDuration="2.971321779s" podCreationTimestamp="2026-05-25 11:19:07 +0000 UTC" firstStartedPulling="2026-05-25 11:19:07.515955687 +0000 UTC m=+87.527564120" lastFinishedPulling="2026-05-25 11:19:08.520521745 +0000 UTC m=+88.532130186" observedRunningTime="2026-05-25 11:19:09.96947644 +0000 UTC m=+89.981084900" watchObservedRunningTime="2026-05-25 11:19:09.971321779 +0000 UTC m=+89.982930236" May 25 11:19:25.912463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:19:25.912435 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-czj9c" May 25 11:22:40.500791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:22:40.500765 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:22:40.501284 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:22:40.500802 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:22:40.503423 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:22:40.503399 2548 kubelet.go:1628] "Image garbage collection succeeded" May 25 11:23:10.150229 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.150200 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-4tbr2"] May 25 11:23:10.153315 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.153299 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.156986 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.156960 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 25 11:23:10.156986 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.156971 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 25 11:23:10.157847 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.157828 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-p6vk4\"" May 25 11:23:10.157930 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.157861 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" May 25 11:23:10.163085 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.163066 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-4tbr2"] May 25 11:23:10.241668 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.241642 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsg92\" (UniqueName: \"kubernetes.io/projected/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-kube-api-access-qsg92\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.241789 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.241688 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-data\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.342899 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.342878 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qsg92\" (UniqueName: \"kubernetes.io/projected/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-kube-api-access-qsg92\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.343017 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.342918 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-data\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.343259 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.343244 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-data\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.352520 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.352502 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsg92\" (UniqueName: \"kubernetes.io/projected/3b9d3ad8-248b-4eb7-b622-f7d30fe35b44-kube-api-access-qsg92\") pod \"seaweedfs-79f457656b-4tbr2\" (UID: \"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44\") " pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.462693 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.462638 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:10.587158 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.587128 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-4tbr2"] May 25 11:23:10.590294 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:23:10.590262 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9d3ad8_248b_4eb7_b622_f7d30fe35b44.slice/crio-4ef1555b4cef6d3c22cf0736a592c240749107fdc2080892ba0bbfc1de3dede0 WatchSource:0}: Error finding container 4ef1555b4cef6d3c22cf0736a592c240749107fdc2080892ba0bbfc1de3dede0: Status 404 returned error can't find the container with id 4ef1555b4cef6d3c22cf0736a592c240749107fdc2080892ba0bbfc1de3dede0 May 25 11:23:10.591827 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:10.591810 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 11:23:11.586688 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:11.586650 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-4tbr2" event={"ID":"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44","Type":"ContainerStarted","Data":"4ef1555b4cef6d3c22cf0736a592c240749107fdc2080892ba0bbfc1de3dede0"} May 25 11:23:13.593039 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:13.592954 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-4tbr2" event={"ID":"3b9d3ad8-248b-4eb7-b622-f7d30fe35b44","Type":"ContainerStarted","Data":"c18a45cba27f1d1f8c671e02cc62a9ab204a567872d02a019b14aec2de8f4976"} May 25 11:23:13.593412 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:13.593122 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:23:13.608412 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:13.608360 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-4tbr2" podStartSLOduration=1.016004114 podStartE2EDuration="3.608342559s" podCreationTimestamp="2026-05-25 11:23:10 +0000 UTC" firstStartedPulling="2026-05-25 11:23:10.59192775 +0000 UTC m=+330.603536182" lastFinishedPulling="2026-05-25 11:23:13.184266185 +0000 UTC m=+333.195874627" observedRunningTime="2026-05-25 11:23:13.607674561 +0000 UTC m=+333.619283028" watchObservedRunningTime="2026-05-25 11:23:13.608342559 +0000 UTC m=+333.619951015" May 25 11:23:19.598427 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:23:19.598391 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-4tbr2" May 25 11:24:20.201667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.201635 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-qwdxm"] May 25 11:24:20.204882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.204861 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.207600 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.207571 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-v6vvh\"" May 25 11:24:20.207720 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.207702 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" May 25 11:24:20.227380 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.227357 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qwdxm"] May 25 11:24:20.296642 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.296621 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c016ee30-d2ed-4386-a9c9-82b758c78e17-cert\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.296773 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.296657 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtx4g\" (UniqueName: \"kubernetes.io/projected/c016ee30-d2ed-4386-a9c9-82b758c78e17-kube-api-access-qtx4g\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.396958 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.396933 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c016ee30-d2ed-4386-a9c9-82b758c78e17-cert\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.397132 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.396991 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtx4g\" (UniqueName: \"kubernetes.io/projected/c016ee30-d2ed-4386-a9c9-82b758c78e17-kube-api-access-qtx4g\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.399510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.399489 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c016ee30-d2ed-4386-a9c9-82b758c78e17-cert\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.407882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.407853 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtx4g\" (UniqueName: \"kubernetes.io/projected/c016ee30-d2ed-4386-a9c9-82b758c78e17-kube-api-access-qtx4g\") pod \"odh-model-controller-696fc77849-qwdxm\" (UID: \"c016ee30-d2ed-4386-a9c9-82b758c78e17\") " pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.514833 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.514767 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:20.638735 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.638706 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qwdxm"] May 25 11:24:20.641743 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:24:20.641712 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc016ee30_d2ed_4386_a9c9_82b758c78e17.slice/crio-27fb3911b10b0d229834018c28917a36c969c5bc84cdcc17de0c9a824b4ad9d9 WatchSource:0}: Error finding container 27fb3911b10b0d229834018c28917a36c969c5bc84cdcc17de0c9a824b4ad9d9: Status 404 returned error can't find the container with id 27fb3911b10b0d229834018c28917a36c969c5bc84cdcc17de0c9a824b4ad9d9 May 25 11:24:20.763926 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:20.763893 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qwdxm" event={"ID":"c016ee30-d2ed-4386-a9c9-82b758c78e17","Type":"ContainerStarted","Data":"27fb3911b10b0d229834018c28917a36c969c5bc84cdcc17de0c9a824b4ad9d9"} May 25 11:24:23.772550 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:23.772514 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qwdxm" event={"ID":"c016ee30-d2ed-4386-a9c9-82b758c78e17","Type":"ContainerStarted","Data":"c7f15d124cb6a42babf0ed7b68b57596df5d86075362fe85b26814533b1d6dbd"} May 25 11:24:23.773162 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:23.772620 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:23.791793 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:23.791745 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-qwdxm" podStartSLOduration=1.214606994 podStartE2EDuration="3.791731909s" podCreationTimestamp="2026-05-25 11:24:20 +0000 UTC" firstStartedPulling="2026-05-25 11:24:20.642905427 +0000 UTC m=+400.654513860" lastFinishedPulling="2026-05-25 11:24:23.220030339 +0000 UTC m=+403.231638775" observedRunningTime="2026-05-25 11:24:23.790554941 +0000 UTC m=+403.802163408" watchObservedRunningTime="2026-05-25 11:24:23.791731909 +0000 UTC m=+403.803340364" May 25 11:24:34.777889 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:34.777860 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-qwdxm" May 25 11:24:54.973270 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.973165 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:24:54.976328 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.976311 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:54.979280 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.979253 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" May 25 11:24:54.979378 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.979279 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 25 11:24:54.979439 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.979412 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 25 11:24:54.980374 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.980354 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chxn5\"" May 25 11:24:54.980442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.980393 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" May 25 11:24:54.987945 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:54.987923 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:24:55.130237 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.130208 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.130435 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.130244 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.130435 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.130276 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.130435 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.130324 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ctf\" (UniqueName: \"kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.155501 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.155469 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:24:55.158714 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.158696 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.161770 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.161749 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21751-predictor-serving-cert\"" May 25 11:24:55.161915 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.161799 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21751-kube-rbac-proxy-sar-config\"" May 25 11:24:55.172983 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.172956 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:24:55.231165 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231068 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.231165 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231123 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.231359 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231284 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ctf\" (UniqueName: \"kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.231411 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231361 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.231462 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231441 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.231694 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.231675 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.234310 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.234285 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.242812 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.242792 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ctf\" (UniqueName: \"kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf\") pod \"isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.277590 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.277555 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:24:55.284189 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.284152 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.287064 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.286662 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:24:55.287064 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.286890 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" May 25 11:24:55.287267 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.287120 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" May 25 11:24:55.289645 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.289627 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:24:55.331877 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.331850 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4drv\" (UniqueName: \"kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.332030 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.331890 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.332030 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.331920 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.427770 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.427749 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:24:55.429919 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:24:55.429893 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83110a4_fc38_4dc8_b29a_e5010f5fa423.slice/crio-2a504a2670644219ad51c612389559aaafa9b3dd4d13763144d772eff3f395c3 WatchSource:0}: Error finding container 2a504a2670644219ad51c612389559aaafa9b3dd4d13763144d772eff3f395c3: Status 404 returned error can't find the container with id 2a504a2670644219ad51c612389559aaafa9b3dd4d13763144d772eff3f395c3 May 25 11:24:55.432382 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432352 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgczb\" (UniqueName: \"kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.432495 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432408 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.432495 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432481 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.432625 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432517 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r4drv\" (UniqueName: \"kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.432625 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432562 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.432625 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432598 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.432777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.432637 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.433704 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.433674 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.435711 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.435686 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.446914 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.446893 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4drv\" (UniqueName: \"kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv\") pod \"success-200-isvc-21751-predictor-5c775cdf48-28pfz\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.469927 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.469902 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:24:55.533617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.533586 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.533746 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.533636 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.533746 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.533670 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vgczb\" (UniqueName: \"kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.533746 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.533727 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.534025 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.533998 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.534474 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.534425 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.536306 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.536284 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.542891 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.542864 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgczb\" (UniqueName: \"kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb\") pod \"isvc-xgboost-graph-predictor-669d8d6456-4wt2l\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.598915 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.598882 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:24:55.601915 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:24:55.601888 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b6e231_c24f_430e_b952_b778e29d1159.slice/crio-d3c08c0843bd9c21630e10aa160306fe4606dd1c9ee5ca46b4a4fc0bd65f2d68 WatchSource:0}: Error finding container d3c08c0843bd9c21630e10aa160306fe4606dd1c9ee5ca46b4a4fc0bd65f2d68: Status 404 returned error can't find the container with id d3c08c0843bd9c21630e10aa160306fe4606dd1c9ee5ca46b4a4fc0bd65f2d68 May 25 11:24:55.611946 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.611927 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:24:55.743882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.743851 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:24:55.747338 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:24:55.747308 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf106cb_fa28_4201_9a8e_1f1a8a6fa055.slice/crio-1cc82a1b1358551e06f968c7b87d220120b97462c745bfc2f3d1d551fdbb1bea WatchSource:0}: Error finding container 1cc82a1b1358551e06f968c7b87d220120b97462c745bfc2f3d1d551fdbb1bea: Status 404 returned error can't find the container with id 1cc82a1b1358551e06f968c7b87d220120b97462c745bfc2f3d1d551fdbb1bea May 25 11:24:55.857257 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.857216 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerStarted","Data":"1cc82a1b1358551e06f968c7b87d220120b97462c745bfc2f3d1d551fdbb1bea"} May 25 11:24:55.858562 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.858534 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerStarted","Data":"d3c08c0843bd9c21630e10aa160306fe4606dd1c9ee5ca46b4a4fc0bd65f2d68"} May 25 11:24:55.859843 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:24:55.859817 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerStarted","Data":"2a504a2670644219ad51c612389559aaafa9b3dd4d13763144d772eff3f395c3"} May 25 11:25:01.886030 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:01.885991 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerStarted","Data":"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66"} May 25 11:25:01.887462 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:01.887437 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerStarted","Data":"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e"} May 25 11:25:05.900947 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:05.900850 2548 generic.go:358] "Generic (PLEG): container finished" podID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerID="8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66" exitCode=0 May 25 11:25:05.900947 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:05.900891 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerDied","Data":"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66"} May 25 11:25:06.912112 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:06.912071 2548 generic.go:358] "Generic (PLEG): container finished" podID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerID="a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e" exitCode=0 May 25 11:25:06.912525 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:06.912143 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerDied","Data":"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e"} May 25 11:25:08.919624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:08.919559 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerStarted","Data":"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29"} May 25 11:25:16.948730 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.948680 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerStarted","Data":"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46"} May 25 11:25:16.949221 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.948840 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:25:16.950848 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.950824 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerStarted","Data":"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e"} May 25 11:25:16.950977 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.950860 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerStarted","Data":"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f"} May 25 11:25:16.951094 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.951075 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:25:16.969488 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.969434 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podStartSLOduration=1.328706232 podStartE2EDuration="21.969417957s" podCreationTimestamp="2026-05-25 11:24:55 +0000 UTC" firstStartedPulling="2026-05-25 11:24:55.603683234 +0000 UTC m=+435.615291666" lastFinishedPulling="2026-05-25 11:25:16.244394948 +0000 UTC m=+456.256003391" observedRunningTime="2026-05-25 11:25:16.968378021 +0000 UTC m=+456.979986491" watchObservedRunningTime="2026-05-25 11:25:16.969417957 +0000 UTC m=+456.981026413" May 25 11:25:16.992258 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:16.992208 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podStartSLOduration=2.178048509 podStartE2EDuration="22.992193777s" podCreationTimestamp="2026-05-25 11:24:54 +0000 UTC" firstStartedPulling="2026-05-25 11:24:55.431784826 +0000 UTC m=+435.443393260" lastFinishedPulling="2026-05-25 11:25:16.245930095 +0000 UTC m=+456.257538528" observedRunningTime="2026-05-25 11:25:16.989062812 +0000 UTC m=+457.000671270" watchObservedRunningTime="2026-05-25 11:25:16.992193777 +0000 UTC m=+457.003802234" May 25 11:25:17.955930 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:17.955889 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:25:17.956405 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:17.955941 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:25:17.956898 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:17.956860 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:17.957022 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:17.956949 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:18.959617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:18.959579 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:18.959617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:18.959597 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:23.965340 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:23.965295 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:25:23.965914 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:23.965482 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:25:23.965914 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:23.965630 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:23.966159 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:23.966134 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:26.985386 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:26.985359 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerStarted","Data":"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b"} May 25 11:25:27.990216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:27.990178 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerStarted","Data":"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e"} May 25 11:25:27.990667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:27.990322 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:25:28.016079 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:28.016034 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podStartSLOduration=1.909078879 podStartE2EDuration="33.016021187s" podCreationTimestamp="2026-05-25 11:24:55 +0000 UTC" firstStartedPulling="2026-05-25 11:24:55.749296664 +0000 UTC m=+435.760905102" lastFinishedPulling="2026-05-25 11:25:26.856238968 +0000 UTC m=+466.867847410" observedRunningTime="2026-05-25 11:25:28.015195436 +0000 UTC m=+468.026803895" watchObservedRunningTime="2026-05-25 11:25:28.016021187 +0000 UTC m=+468.027629642" May 25 11:25:28.993169 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:28.993132 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:25:28.994285 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:28.994255 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:25:29.996829 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:29.996789 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:25:33.965993 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:33.965951 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:33.966413 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:33.966120 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:35.001463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:35.001435 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:25:35.002061 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:35.002036 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:25:43.966682 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:43.966634 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:43.967064 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:43.966634 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:45.002143 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:45.002085 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:25:53.966657 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:53.966613 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused" May 25 11:25:53.967138 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:53.966613 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:25:55.002640 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:25:55.002607 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:26:03.965881 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:03.965839 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:26:03.966449 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:03.966249 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:26:05.002912 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:05.002873 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:26:13.966739 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:13.966685 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused" May 25 11:26:14.963305 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:14.963268 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:14.991404 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:14.991372 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:14.991754 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:14.991499 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:14.994270 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:14.994247 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-21751-serving-cert\"" May 25 11:26:14.994390 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:14.994269 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-21751-kube-rbac-proxy-sar-config\"" May 25 11:26:15.002049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.002019 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 25 11:26:15.068762 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.068736 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.068895 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.068779 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.169636 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.169601 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.169778 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.169647 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.170234 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.170210 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.172141 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.172084 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls\") pod \"switch-graph-21751-5984bd7768-fbhlq\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.303333 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.303268 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:15.428233 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:15.428202 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:15.431402 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:26:15.431375 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e728b7_eaf4_4c88_8b71_c679395cec9c.slice/crio-e6b72a870cd2453f7849e69b86c330331989280508f37363d198c8c8e65de5fb WatchSource:0}: Error finding container e6b72a870cd2453f7849e69b86c330331989280508f37363d198c8c8e65de5fb: Status 404 returned error can't find the container with id e6b72a870cd2453f7849e69b86c330331989280508f37363d198c8c8e65de5fb May 25 11:26:16.116266 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:16.116227 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" event={"ID":"d9e728b7-eaf4-4c88-8b71-c679395cec9c","Type":"ContainerStarted","Data":"e6b72a870cd2453f7849e69b86c330331989280508f37363d198c8c8e65de5fb"} May 25 11:26:18.124021 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:18.123988 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" event={"ID":"d9e728b7-eaf4-4c88-8b71-c679395cec9c","Type":"ContainerStarted","Data":"feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635"} May 25 11:26:18.124418 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:18.124136 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:18.143481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:18.143439 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podStartSLOduration=1.846358241 podStartE2EDuration="4.143424874s" podCreationTimestamp="2026-05-25 11:26:14 +0000 UTC" firstStartedPulling="2026-05-25 11:26:15.433474639 +0000 UTC m=+515.445083072" lastFinishedPulling="2026-05-25 11:26:17.730541268 +0000 UTC m=+517.742149705" observedRunningTime="2026-05-25 11:26:18.141576226 +0000 UTC m=+518.153184692" watchObservedRunningTime="2026-05-25 11:26:18.143424874 +0000 UTC m=+518.155033329" May 25 11:26:23.966272 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:23.966233 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:26:24.132520 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:24.132495 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:25.002964 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.002933 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:26:25.139846 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.139818 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:25.140042 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.140019 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" containerID="cri-o://feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635" gracePeriod=30 May 25 11:26:25.251003 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.250976 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:26:25.251280 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.251257 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" containerID="cri-o://1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29" gracePeriod=30 May 25 11:26:25.251381 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.251299 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kube-rbac-proxy" containerID="cri-o://85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46" gracePeriod=30 May 25 11:26:25.328337 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.328303 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:26:25.331691 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.331669 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.333768 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.333747 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-55329-predictor-serving-cert\"" May 25 11:26:25.334186 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.334163 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-55329-kube-rbac-proxy-sar-config\"" May 25 11:26:25.355684 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.353670 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.355684 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.353727 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x467p\" (UniqueName: \"kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.355684 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.353817 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.356610 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.356472 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:26:25.454792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.454760 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.454943 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.454800 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x467p\" (UniqueName: \"kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.454943 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.454861 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.455060 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:25.454989 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-55329-predictor-serving-cert: secret "success-200-isvc-55329-predictor-serving-cert" not found May 25 11:26:25.455060 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:25.455056 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls podName:91b69d67-be24-495e-a758-f093947a9b5b nodeName:}" failed. No retries permitted until 2026-05-25 11:26:25.955036776 +0000 UTC m=+525.966645408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls") pod "success-200-isvc-55329-predictor-867dcfc56d-6sgzv" (UID: "91b69d67-be24-495e-a758-f093947a9b5b") : secret "success-200-isvc-55329-predictor-serving-cert" not found May 25 11:26:25.455536 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.455509 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.466954 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.466927 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x467p\" (UniqueName: \"kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.958171 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.958139 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:25.960577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:25.960556 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") pod \"success-200-isvc-55329-predictor-867dcfc56d-6sgzv\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:26.148043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:26.148014 2548 generic.go:358] "Generic (PLEG): container finished" podID="55b6e231-c24f-430e-b952-b778e29d1159" containerID="85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46" exitCode=2 May 25 11:26:26.148412 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:26.148057 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerDied","Data":"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46"} May 25 11:26:26.255666 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:26.255583 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:26.379083 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:26.379049 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:26:26.381965 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:26:26.381924 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b69d67_be24_495e_a758_f093947a9b5b.slice/crio-ae42221597d4a50b429971f59ca33c2c6521ebfdfda3d5971db5d0e367fc4966 WatchSource:0}: Error finding container ae42221597d4a50b429971f59ca33c2c6521ebfdfda3d5971db5d0e367fc4966: Status 404 returned error can't find the container with id ae42221597d4a50b429971f59ca33c2c6521ebfdfda3d5971db5d0e367fc4966 May 25 11:26:27.153209 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:27.153170 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerStarted","Data":"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021"} May 25 11:26:27.153209 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:27.153207 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerStarted","Data":"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24"} May 25 11:26:27.153209 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:27.153216 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerStarted","Data":"ae42221597d4a50b429971f59ca33c2c6521ebfdfda3d5971db5d0e367fc4966"} May 25 11:26:27.153647 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:27.153302 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:27.172791 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:27.172746 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podStartSLOduration=2.172732686 podStartE2EDuration="2.172732686s" podCreationTimestamp="2026-05-25 11:26:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:26:27.171526043 +0000 UTC m=+527.183134499" watchObservedRunningTime="2026-05-25 11:26:27.172732686 +0000 UTC m=+527.184341141" May 25 11:26:28.101482 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.101462 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:26:28.157510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157483 2548 generic.go:358] "Generic (PLEG): container finished" podID="55b6e231-c24f-430e-b952-b778e29d1159" containerID="1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29" exitCode=0 May 25 11:26:28.157858 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157574 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" May 25 11:26:28.157858 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157573 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerDied","Data":"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29"} May 25 11:26:28.157858 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157610 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz" event={"ID":"55b6e231-c24f-430e-b952-b778e29d1159","Type":"ContainerDied","Data":"d3c08c0843bd9c21630e10aa160306fe4606dd1c9ee5ca46b4a4fc0bd65f2d68"} May 25 11:26:28.157858 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157628 2548 scope.go:117] "RemoveContainer" containerID="85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46" May 25 11:26:28.158046 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.157996 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:28.159223 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.159198 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:26:28.165273 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.165256 2548 scope.go:117] "RemoveContainer" containerID="1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29" May 25 11:26:28.171862 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.171847 2548 scope.go:117] "RemoveContainer" containerID="85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46" May 25 11:26:28.172193 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:28.172087 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46\": container with ID starting with 85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46 not found: ID does not exist" containerID="85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46" May 25 11:26:28.172255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.172203 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46"} err="failed to get container status \"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46\": rpc error: code = NotFound desc = could not find container \"85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46\": container with ID starting with 85566e545f37c8cee24d930a614abef1b2664efe70a54deeb2c34cc055d42e46 not found: ID does not exist" May 25 11:26:28.172255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.172232 2548 scope.go:117] "RemoveContainer" containerID="1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29" May 25 11:26:28.172451 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:28.172436 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29\": container with ID starting with 1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29 not found: ID does not exist" containerID="1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29" May 25 11:26:28.172491 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.172455 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29"} err="failed to get container status \"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29\": rpc error: code = NotFound desc = could not find container \"1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29\": container with ID starting with 1a638db157bd18cb19ec23d539f49876790866230d23a6a426d63a030eb88f29 not found: ID does not exist" May 25 11:26:28.174673 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.174659 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4drv\" (UniqueName: \"kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv\") pod \"55b6e231-c24f-430e-b952-b778e29d1159\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " May 25 11:26:28.174715 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.174709 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls\") pod \"55b6e231-c24f-430e-b952-b778e29d1159\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " May 25 11:26:28.174748 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.174738 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config\") pod \"55b6e231-c24f-430e-b952-b778e29d1159\" (UID: \"55b6e231-c24f-430e-b952-b778e29d1159\") " May 25 11:26:28.175065 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.175044 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-21751-kube-rbac-proxy-sar-config") pod "55b6e231-c24f-430e-b952-b778e29d1159" (UID: "55b6e231-c24f-430e-b952-b778e29d1159"). InnerVolumeSpecName "success-200-isvc-21751-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:26:28.176755 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.176737 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55b6e231-c24f-430e-b952-b778e29d1159" (UID: "55b6e231-c24f-430e-b952-b778e29d1159"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:26:28.176755 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.176746 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv" (OuterVolumeSpecName: "kube-api-access-r4drv") pod "55b6e231-c24f-430e-b952-b778e29d1159" (UID: "55b6e231-c24f-430e-b952-b778e29d1159"). InnerVolumeSpecName "kube-api-access-r4drv". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:26:28.275836 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.275783 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55b6e231-c24f-430e-b952-b778e29d1159-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:28.275836 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.275803 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55b6e231-c24f-430e-b952-b778e29d1159-success-200-isvc-21751-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:28.275836 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.275813 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4drv\" (UniqueName: \"kubernetes.io/projected/55b6e231-c24f-430e-b952-b778e29d1159-kube-api-access-r4drv\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:28.479139 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.479095 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:26:28.482513 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.482491 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21751-predictor-5c775cdf48-28pfz"] May 25 11:26:28.599664 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:28.599602 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b6e231-c24f-430e-b952-b778e29d1159" path="/var/lib/kubelet/pods/55b6e231-c24f-430e-b952-b778e29d1159/volumes" May 25 11:26:29.130825 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:29.130788 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:29.161171 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:29.161136 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:26:34.131020 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:34.130973 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:34.165027 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:34.165000 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:26:34.165584 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:34.165554 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:26:39.131224 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:39.131182 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:39.131678 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:39.131286 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:44.132039 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:44.131987 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:44.165830 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:44.165798 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:26:49.131648 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:49.131611 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:54.131667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:54.131617 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:26:54.166567 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:54.166533 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:26:55.087014 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.086984 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:26:55.087347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.087315 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" containerID="cri-o://29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b" gracePeriod=30 May 25 11:26:55.087481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.087366 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kube-rbac-proxy" containerID="cri-o://a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e" gracePeriod=30 May 25 11:26:55.171181 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:55.171150 2548 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e728b7_eaf4_4c88_8b71_c679395cec9c.slice/crio-conmon-feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635.scope\": RecentStats: unable to find data in memory cache]" May 25 11:26:55.191826 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.191800 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:26:55.192207 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192184 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kube-rbac-proxy" May 25 11:26:55.192207 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192204 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kube-rbac-proxy" May 25 11:26:55.192374 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192238 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" May 25 11:26:55.192374 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192246 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" May 25 11:26:55.192374 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192314 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kube-rbac-proxy" May 25 11:26:55.192374 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.192327 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="55b6e231-c24f-430e-b952-b778e29d1159" containerName="kserve-container" May 25 11:26:55.196396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.196375 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.199317 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.199117 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f16f5-predictor-serving-cert\"" May 25 11:26:55.199317 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.199266 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\"" May 25 11:26:55.205003 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.204962 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:26:55.210917 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.210590 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:26:55.211013 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.210929 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" containerID="cri-o://136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f" gracePeriod=30 May 25 11:26:55.211079 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.211014 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kube-rbac-proxy" containerID="cri-o://0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e" gracePeriod=30 May 25 11:26:55.243717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.241028 2548 generic.go:358] "Generic (PLEG): container finished" podID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerID="feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635" exitCode=0 May 25 11:26:55.243717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.241210 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" event={"ID":"d9e728b7-eaf4-4c88-8b71-c679395cec9c","Type":"ContainerDied","Data":"feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635"} May 25 11:26:55.243717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.243076 2548 generic.go:358] "Generic (PLEG): container finished" podID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerID="a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e" exitCode=2 May 25 11:26:55.243717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.243133 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerDied","Data":"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e"} May 25 11:26:55.264340 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.264258 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.264340 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.264296 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.264520 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.264355 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpv4m\" (UniqueName: \"kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.351188 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.351156 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:55.365934 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.365906 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.366059 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.365952 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.366059 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.366015 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jpv4m\" (UniqueName: \"kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.366205 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:55.366088 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-serving-cert: secret "success-200-isvc-f16f5-predictor-serving-cert" not found May 25 11:26:55.366205 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:55.366181 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls podName:d694fff0-c01e-41f2-8e8f-0fb8fb79cb97 nodeName:}" failed. No retries permitted until 2026-05-25 11:26:55.866160478 +0000 UTC m=+555.877768925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls") pod "success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" (UID: "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97") : secret "success-200-isvc-f16f5-predictor-serving-cert" not found May 25 11:26:55.366740 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.366712 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.374740 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.374714 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpv4m\" (UniqueName: \"kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.467006 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.466954 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle\") pod \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " May 25 11:26:55.467221 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.467080 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls\") pod \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\" (UID: \"d9e728b7-eaf4-4c88-8b71-c679395cec9c\") " May 25 11:26:55.467343 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.467318 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d9e728b7-eaf4-4c88-8b71-c679395cec9c" (UID: "d9e728b7-eaf4-4c88-8b71-c679395cec9c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:26:55.469329 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.469299 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9e728b7-eaf4-4c88-8b71-c679395cec9c" (UID: "d9e728b7-eaf4-4c88-8b71-c679395cec9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:26:55.568329 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.568282 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9e728b7-eaf4-4c88-8b71-c679395cec9c-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:55.568329 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.568337 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e728b7-eaf4-4c88-8b71-c679395cec9c-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:55.869826 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.869789 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:55.872288 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:55.872269 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") pod \"success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:56.128347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.128283 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:56.246814 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.246783 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" event={"ID":"d9e728b7-eaf4-4c88-8b71-c679395cec9c","Type":"ContainerDied","Data":"e6b72a870cd2453f7849e69b86c330331989280508f37363d198c8c8e65de5fb"} May 25 11:26:56.246814 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.246805 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq" May 25 11:26:56.247305 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.246835 2548 scope.go:117] "RemoveContainer" containerID="feab9e46bdecdc2f889ac60591e48498b0c1a004a03a34f4ed3b71ec151da635" May 25 11:26:56.248897 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.248864 2548 generic.go:358] "Generic (PLEG): container finished" podID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerID="0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e" exitCode=2 May 25 11:26:56.249005 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.248922 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerDied","Data":"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e"} May 25 11:26:56.253644 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.253587 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:26:56.257616 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:26:56.257597 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd694fff0_c01e_41f2_8e8f_0fb8fb79cb97.slice/crio-86f523dc160eaa299b0cf1253deba04b05aa9daa53f3a4be17a9c95e0303eacf WatchSource:0}: Error finding container 86f523dc160eaa299b0cf1253deba04b05aa9daa53f3a4be17a9c95e0303eacf: Status 404 returned error can't find the container with id 86f523dc160eaa299b0cf1253deba04b05aa9daa53f3a4be17a9c95e0303eacf May 25 11:26:56.270179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.270159 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:56.273950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.273930 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-21751-5984bd7768-fbhlq"] May 25 11:26:56.598473 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:56.598439 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" path="/var/lib/kubelet/pods/d9e728b7-eaf4-4c88-8b71-c679395cec9c/volumes" May 25 11:26:57.252879 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:57.252838 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerStarted","Data":"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34"} May 25 11:26:57.253354 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:57.252884 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerStarted","Data":"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3"} May 25 11:26:57.253354 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:57.252901 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerStarted","Data":"86f523dc160eaa299b0cf1253deba04b05aa9daa53f3a4be17a9c95e0303eacf"} May 25 11:26:57.253354 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:57.253023 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:57.273607 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:57.273560 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podStartSLOduration=2.273546595 podStartE2EDuration="2.273546595s" podCreationTimestamp="2026-05-25 11:26:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:26:57.272608953 +0000 UTC m=+557.284217410" watchObservedRunningTime="2026-05-25 11:26:57.273546595 +0000 UTC m=+557.285155049" May 25 11:26:58.256657 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.256625 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:26:58.257838 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.257808 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:26:58.737621 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.737589 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:26:58.897684 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.897653 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location\") pod \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " May 25 11:26:58.897861 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.897692 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " May 25 11:26:58.897861 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.897720 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgczb\" (UniqueName: \"kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb\") pod \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " May 25 11:26:58.897861 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.897747 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls\") pod \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\" (UID: \"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055\") " May 25 11:26:58.898046 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.897983 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" (UID: "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 11:26:58.898128 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.898057 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" (UID: "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:26:58.899888 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.899867 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb" (OuterVolumeSpecName: "kube-api-access-vgczb") pod "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" (UID: "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055"). InnerVolumeSpecName "kube-api-access-vgczb". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:26:58.899963 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.899945 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" (UID: "6bf106cb-fa28-4201-9a8e-1f1a8a6fa055"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:26:58.960710 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.960666 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused" May 25 11:26:58.999038 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.999008 2548 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kserve-provision-location\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:58.999038 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.999033 2548 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:58.999226 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.999048 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgczb\" (UniqueName: \"kubernetes.io/projected/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-kube-api-access-vgczb\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:58.999226 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:58.999062 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:59.261347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261318 2548 generic.go:358] "Generic (PLEG): container finished" podID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerID="29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b" exitCode=0 May 25 11:26:59.261717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261397 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerDied","Data":"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b"} May 25 11:26:59.261717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261413 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" May 25 11:26:59.261717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261439 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l" event={"ID":"6bf106cb-fa28-4201-9a8e-1f1a8a6fa055","Type":"ContainerDied","Data":"1cc82a1b1358551e06f968c7b87d220120b97462c745bfc2f3d1d551fdbb1bea"} May 25 11:26:59.261717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261454 2548 scope.go:117] "RemoveContainer" containerID="a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e" May 25 11:26:59.261967 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.261816 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:26:59.311347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.311316 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:26:59.318678 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.318655 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-4wt2l"] May 25 11:26:59.323586 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.323570 2548 scope.go:117] "RemoveContainer" containerID="29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b" May 25 11:26:59.330292 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.330258 2548 scope.go:117] "RemoveContainer" containerID="8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66" May 25 11:26:59.338492 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.338468 2548 scope.go:117] "RemoveContainer" containerID="a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e" May 25 11:26:59.338801 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:59.338776 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e\": container with ID starting with a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e not found: ID does not exist" containerID="a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e" May 25 11:26:59.338885 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.338815 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e"} err="failed to get container status \"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e\": rpc error: code = NotFound desc = could not find container \"a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e\": container with ID starting with a84c75683f5e9bc433aaa70f1fea1c02bc1ed44975da5aafa9d10c6d859e4c8e not found: ID does not exist" May 25 11:26:59.338885 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.338844 2548 scope.go:117] "RemoveContainer" containerID="29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b" May 25 11:26:59.339436 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:59.339293 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b\": container with ID starting with 29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b not found: ID does not exist" containerID="29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b" May 25 11:26:59.339436 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.339334 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b"} err="failed to get container status \"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b\": rpc error: code = NotFound desc = could not find container \"29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b\": container with ID starting with 29ea6cdf1a4cdecf86e52f931f7fc98409ca82a6b94ab70cfa7208f4168ec52b not found: ID does not exist" May 25 11:26:59.339436 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.339357 2548 scope.go:117] "RemoveContainer" containerID="8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66" May 25 11:26:59.340845 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:26:59.340806 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66\": container with ID starting with 8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66 not found: ID does not exist" containerID="8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66" May 25 11:26:59.340939 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.340852 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66"} err="failed to get container status \"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66\": rpc error: code = NotFound desc = could not find container \"8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66\": container with ID starting with 8823328d5a9c1cdc57a9ad11203ae842ce314c923cb2619288b6e14a0718cb66 not found: ID does not exist" May 25 11:26:59.447464 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.447444 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:26:59.603991 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.603966 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location\") pod \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " May 25 11:26:59.604126 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.604013 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls\") pod \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " May 25 11:26:59.604126 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.604037 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " May 25 11:26:59.604126 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.604083 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9ctf\" (UniqueName: \"kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf\") pod \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\" (UID: \"f83110a4-fc38-4dc8-b29a-e5010f5fa423\") " May 25 11:26:59.604370 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.604347 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f83110a4-fc38-4dc8-b29a-e5010f5fa423" (UID: "f83110a4-fc38-4dc8-b29a-e5010f5fa423"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 11:26:59.604446 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.604427 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "f83110a4-fc38-4dc8-b29a-e5010f5fa423" (UID: "f83110a4-fc38-4dc8-b29a-e5010f5fa423"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:26:59.606217 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.606192 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf" (OuterVolumeSpecName: "kube-api-access-t9ctf") pod "f83110a4-fc38-4dc8-b29a-e5010f5fa423" (UID: "f83110a4-fc38-4dc8-b29a-e5010f5fa423"). InnerVolumeSpecName "kube-api-access-t9ctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:26:59.606217 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.606199 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f83110a4-fc38-4dc8-b29a-e5010f5fa423" (UID: "f83110a4-fc38-4dc8-b29a-e5010f5fa423"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:26:59.705232 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.705207 2548 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f83110a4-fc38-4dc8-b29a-e5010f5fa423-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:59.705232 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.705230 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9ctf\" (UniqueName: \"kubernetes.io/projected/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kube-api-access-t9ctf\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:59.705368 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.705240 2548 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f83110a4-fc38-4dc8-b29a-e5010f5fa423-kserve-provision-location\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:26:59.705368 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:26:59.705249 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f83110a4-fc38-4dc8-b29a-e5010f5fa423-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:27:00.266608 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.266570 2548 generic.go:358] "Generic (PLEG): container finished" podID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerID="136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f" exitCode=0 May 25 11:27:00.267031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.266658 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" May 25 11:27:00.267031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.266660 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerDied","Data":"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f"} May 25 11:27:00.267031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.266706 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt" event={"ID":"f83110a4-fc38-4dc8-b29a-e5010f5fa423","Type":"ContainerDied","Data":"2a504a2670644219ad51c612389559aaafa9b3dd4d13763144d772eff3f395c3"} May 25 11:27:00.267031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.266737 2548 scope.go:117] "RemoveContainer" containerID="0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e" May 25 11:27:00.274610 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.274581 2548 scope.go:117] "RemoveContainer" containerID="136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f" May 25 11:27:00.281532 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.281516 2548 scope.go:117] "RemoveContainer" containerID="a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e" May 25 11:27:00.287085 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.287066 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:27:00.288501 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.288486 2548 scope.go:117] "RemoveContainer" containerID="0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e" May 25 11:27:00.288754 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:27:00.288735 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e\": container with ID starting with 0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e not found: ID does not exist" containerID="0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e" May 25 11:27:00.288812 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.288759 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e"} err="failed to get container status \"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e\": rpc error: code = NotFound desc = could not find container \"0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e\": container with ID starting with 0c7fd4ecf8d056573478fb3a1a7e31c768165e95c0635f076fcccbc1f5a68e4e not found: ID does not exist" May 25 11:27:00.288812 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.288777 2548 scope.go:117] "RemoveContainer" containerID="136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f" May 25 11:27:00.288992 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:27:00.288976 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f\": container with ID starting with 136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f not found: ID does not exist" containerID="136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f" May 25 11:27:00.289032 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.288998 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f"} err="failed to get container status \"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f\": rpc error: code = NotFound desc = could not find container \"136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f\": container with ID starting with 136d6b177161ce8d4be3c92aeedc8123d5e57e1bcbc11b166e10c890ef1ea82f not found: ID does not exist" May 25 11:27:00.289032 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.289013 2548 scope.go:117] "RemoveContainer" containerID="a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e" May 25 11:27:00.289228 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:27:00.289213 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e\": container with ID starting with a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e not found: ID does not exist" containerID="a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e" May 25 11:27:00.289274 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.289231 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e"} err="failed to get container status \"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e\": rpc error: code = NotFound desc = could not find container \"a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e\": container with ID starting with a99fd0b2af0c4d8bf71010c614e8351d8fa77bce876db5d33a6d9f658307ea5e not found: ID does not exist" May 25 11:27:00.292491 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.292471 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5474d9d7fb-c7tvt"] May 25 11:27:00.598659 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.598588 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" path="/var/lib/kubelet/pods/6bf106cb-fa28-4201-9a8e-1f1a8a6fa055/volumes" May 25 11:27:00.599048 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:00.599036 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" path="/var/lib/kubelet/pods/f83110a4-fc38-4dc8-b29a-e5010f5fa423/volumes" May 25 11:27:04.165718 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:04.165677 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 25 11:27:04.265986 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:04.265955 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:27:04.266415 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:04.266384 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:27:14.166958 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:14.166923 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:27:14.266835 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:14.266790 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:27:24.267260 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:24.267215 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:27:25.373468 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373434 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:27:25.373854 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373839 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="storage-initializer" May 25 11:27:25.373896 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373856 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="storage-initializer" May 25 11:27:25.373896 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373873 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" May 25 11:27:25.373896 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373882 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373904 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373912 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373920 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kube-rbac-proxy" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373928 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kube-rbac-proxy" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373939 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="storage-initializer" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373946 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="storage-initializer" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373959 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373967 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373977 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kube-rbac-proxy" May 25 11:27:25.373989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.373985 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kube-rbac-proxy" May 25 11:27:25.374334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.374036 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kube-rbac-proxy" May 25 11:27:25.374334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.374046 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="f83110a4-fc38-4dc8-b29a-e5010f5fa423" containerName="kserve-container" May 25 11:27:25.374334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.374055 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kube-rbac-proxy" May 25 11:27:25.374334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.374066 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="6bf106cb-fa28-4201-9a8e-1f1a8a6fa055" containerName="kserve-container" May 25 11:27:25.374334 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.374074 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d9e728b7-eaf4-4c88-8b71-c679395cec9c" containerName="switch-graph-21751" May 25 11:27:25.377144 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.377126 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.379274 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.379251 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-55329-kube-rbac-proxy-sar-config\"" May 25 11:27:25.379403 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.379275 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-55329-serving-cert\"" May 25 11:27:25.382815 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.382790 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:27:25.388021 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.387992 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.388154 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.388055 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.489420 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.489396 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.489533 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.489439 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.489577 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:27:25.489534 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-55329-serving-cert: secret "switch-graph-55329-serving-cert" not found May 25 11:27:25.489619 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:27:25.489605 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls podName:c93fcb61-4aaf-4d32-a09a-1f8a2cc18457 nodeName:}" failed. No retries permitted until 2026-05-25 11:27:25.989586824 +0000 UTC m=+586.001195257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls") pod "switch-graph-55329-668cc95c65-m2767" (UID: "c93fcb61-4aaf-4d32-a09a-1f8a2cc18457") : secret "switch-graph-55329-serving-cert" not found May 25 11:27:25.489988 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.489972 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.992069 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.992040 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:25.994612 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:25.994593 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") pod \"switch-graph-55329-668cc95c65-m2767\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:26.288396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:26.288326 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:26.412014 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:26.411985 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:27:26.414979 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:27:26.414948 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93fcb61_4aaf_4d32_a09a_1f8a2cc18457.slice/crio-cdf2b2faf6f3ebeb9f8f42d60a21caab86faacd52525d8a6f039c1a57c0ad1b2 WatchSource:0}: Error finding container cdf2b2faf6f3ebeb9f8f42d60a21caab86faacd52525d8a6f039c1a57c0ad1b2: Status 404 returned error can't find the container with id cdf2b2faf6f3ebeb9f8f42d60a21caab86faacd52525d8a6f039c1a57c0ad1b2 May 25 11:27:27.345701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:27.345667 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" event={"ID":"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457","Type":"ContainerStarted","Data":"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd"} May 25 11:27:27.345701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:27.345702 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" event={"ID":"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457","Type":"ContainerStarted","Data":"cdf2b2faf6f3ebeb9f8f42d60a21caab86faacd52525d8a6f039c1a57c0ad1b2"} May 25 11:27:27.345864 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:27.345797 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:27.369296 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:27.369258 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podStartSLOduration=2.369245335 podStartE2EDuration="2.369245335s" podCreationTimestamp="2026-05-25 11:27:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:27:27.367569017 +0000 UTC m=+587.379177473" watchObservedRunningTime="2026-05-25 11:27:27.369245335 +0000 UTC m=+587.380853789" May 25 11:27:33.354423 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:33.354385 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:27:34.266618 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:34.266580 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 25 11:27:40.519265 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:40.519240 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:27:40.520882 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:40.520863 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:27:44.267892 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:27:44.267864 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:32:40.540259 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:32:40.540224 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:32:40.541937 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:32:40.541910 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:35:40.312367 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.312331 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:35:40.314795 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.312589 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" containerID="cri-o://b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd" gracePeriod=30 May 25 11:35:40.704291 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.704256 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:35:40.704586 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.704539 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" containerID="cri-o://418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24" gracePeriod=30 May 25 11:35:40.704724 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.704579 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kube-rbac-proxy" containerID="cri-o://36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021" gracePeriod=30 May 25 11:35:40.931197 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.931160 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:35:40.934697 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.934677 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:40.941827 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.941805 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49543-predictor-serving-cert\"" May 25 11:35:40.941939 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.941915 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49543-kube-rbac-proxy-sar-config\"" May 25 11:35:40.990057 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.989983 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rhv\" (UniqueName: \"kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:40.990057 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.990018 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:40.990057 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:40.990040 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.021211 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.021179 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:35:41.090868 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.090837 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rhv\" (UniqueName: \"kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.091031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.090880 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.091031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.090904 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.091589 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.091569 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.093499 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.093475 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.114963 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.114935 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rhv\" (UniqueName: \"kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv\") pod \"success-200-isvc-49543-predictor-75fc554798-2v6w6\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.245157 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.245048 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.388265 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.388052 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:35:41.390744 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:35:41.390712 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2411ec6_a6aa_478a_8042_b43378fb3912.slice/crio-931fa21500b7872fe065ab828cc7ba27d3ce80dd95e85254e3c63b8891a26c21 WatchSource:0}: Error finding container 931fa21500b7872fe065ab828cc7ba27d3ce80dd95e85254e3c63b8891a26c21: Status 404 returned error can't find the container with id 931fa21500b7872fe065ab828cc7ba27d3ce80dd95e85254e3c63b8891a26c21 May 25 11:35:41.392428 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.392412 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 11:35:41.658199 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.658157 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerStarted","Data":"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348"} May 25 11:35:41.658199 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.658205 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerStarted","Data":"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a"} May 25 11:35:41.658468 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.658218 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerStarted","Data":"931fa21500b7872fe065ab828cc7ba27d3ce80dd95e85254e3c63b8891a26c21"} May 25 11:35:41.658468 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.658369 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.658580 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.658505 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:41.659825 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.659792 2548 generic.go:358] "Generic (PLEG): container finished" podID="91b69d67-be24-495e-a758-f093947a9b5b" containerID="36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021" exitCode=2 May 25 11:35:41.659939 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.659847 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerDied","Data":"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021"} May 25 11:35:41.659939 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:41.659887 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:35:42.662611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:42.662571 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:35:43.352535 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:43.352500 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:35:43.947400 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:43.947377 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:35:43.984432 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:43.984339 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podStartSLOduration=3.984324598 podStartE2EDuration="3.984324598s" podCreationTimestamp="2026-05-25 11:35:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:35:41.711419538 +0000 UTC m=+1081.723027994" watchObservedRunningTime="2026-05-25 11:35:43.984324598 +0000 UTC m=+1083.995933057" May 25 11:35:44.011355 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.011329 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") pod \"91b69d67-be24-495e-a758-f093947a9b5b\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " May 25 11:35:44.011485 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.011380 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config\") pod \"91b69d67-be24-495e-a758-f093947a9b5b\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " May 25 11:35:44.011544 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.011489 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x467p\" (UniqueName: \"kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p\") pod \"91b69d67-be24-495e-a758-f093947a9b5b\" (UID: \"91b69d67-be24-495e-a758-f093947a9b5b\") " May 25 11:35:44.011725 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.011703 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-55329-kube-rbac-proxy-sar-config") pod "91b69d67-be24-495e-a758-f093947a9b5b" (UID: "91b69d67-be24-495e-a758-f093947a9b5b"). InnerVolumeSpecName "success-200-isvc-55329-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:35:44.013577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.013554 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91b69d67-be24-495e-a758-f093947a9b5b" (UID: "91b69d67-be24-495e-a758-f093947a9b5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:35:44.013650 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.013631 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p" (OuterVolumeSpecName: "kube-api-access-x467p") pod "91b69d67-be24-495e-a758-f093947a9b5b" (UID: "91b69d67-be24-495e-a758-f093947a9b5b"). InnerVolumeSpecName "kube-api-access-x467p". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:35:44.112151 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.112116 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x467p\" (UniqueName: \"kubernetes.io/projected/91b69d67-be24-495e-a758-f093947a9b5b-kube-api-access-x467p\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:35:44.112151 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.112151 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91b69d67-be24-495e-a758-f093947a9b5b-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:35:44.112151 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.112163 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91b69d67-be24-495e-a758-f093947a9b5b-success-200-isvc-55329-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:35:44.668918 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.668888 2548 generic.go:358] "Generic (PLEG): container finished" podID="91b69d67-be24-495e-a758-f093947a9b5b" containerID="418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24" exitCode=0 May 25 11:35:44.669071 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.668959 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" May 25 11:35:44.669071 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.668962 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerDied","Data":"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24"} May 25 11:35:44.669071 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.669066 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv" event={"ID":"91b69d67-be24-495e-a758-f093947a9b5b","Type":"ContainerDied","Data":"ae42221597d4a50b429971f59ca33c2c6521ebfdfda3d5971db5d0e367fc4966"} May 25 11:35:44.669235 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.669082 2548 scope.go:117] "RemoveContainer" containerID="36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021" May 25 11:35:44.676716 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.676697 2548 scope.go:117] "RemoveContainer" containerID="418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24" May 25 11:35:44.683636 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.683620 2548 scope.go:117] "RemoveContainer" containerID="36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021" May 25 11:35:44.683922 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:35:44.683898 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021\": container with ID starting with 36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021 not found: ID does not exist" containerID="36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021" May 25 11:35:44.684033 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.683934 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021"} err="failed to get container status \"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021\": rpc error: code = NotFound desc = could not find container \"36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021\": container with ID starting with 36cdab30f86fc8e334433b9815b24df3e953e10272f8e6259d47c889dbba7021 not found: ID does not exist" May 25 11:35:44.684033 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.683957 2548 scope.go:117] "RemoveContainer" containerID="418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24" May 25 11:35:44.684301 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:35:44.684270 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24\": container with ID starting with 418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24 not found: ID does not exist" containerID="418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24" May 25 11:35:44.684361 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.684306 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24"} err="failed to get container status \"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24\": rpc error: code = NotFound desc = could not find container \"418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24\": container with ID starting with 418c13204775e9581a9284bad5d17b952c26451ed6d773db09c8fa204f0a9d24 not found: ID does not exist" May 25 11:35:44.699292 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.699266 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:35:44.707074 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:44.707051 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55329-predictor-867dcfc56d-6sgzv"] May 25 11:35:46.599050 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:46.599017 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b69d67-be24-495e-a758-f093947a9b5b" path="/var/lib/kubelet/pods/91b69d67-be24-495e-a758-f093947a9b5b/volumes" May 25 11:35:47.666827 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:47.666800 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:35:47.667295 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:47.667228 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:35:48.352812 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:48.352770 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:35:53.353001 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:53.352964 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:35:53.353406 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:53.353118 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:35:57.667745 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:57.667706 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:35:58.353525 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:35:58.353487 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:36:03.352178 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:03.352141 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:36:07.667670 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:07.667627 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:36:08.353202 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:08.353161 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:36:10.051398 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.051369 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:36:10.051809 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.051622 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" containerID="cri-o://fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3" gracePeriod=30 May 25 11:36:10.051809 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.051718 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kube-rbac-proxy" containerID="cri-o://5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34" gracePeriod=30 May 25 11:36:10.088293 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088262 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:36:10.088542 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088529 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kube-rbac-proxy" May 25 11:36:10.088583 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088544 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kube-rbac-proxy" May 25 11:36:10.088583 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088552 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" May 25 11:36:10.088583 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088557 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" May 25 11:36:10.088667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088622 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kserve-container" May 25 11:36:10.088667 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.088630 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="91b69d67-be24-495e-a758-f093947a9b5b" containerName="kube-rbac-proxy" May 25 11:36:10.091510 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.091495 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.096931 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.096909 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-412bf-predictor-serving-cert\"" May 25 11:36:10.114455 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.114433 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-412bf-kube-rbac-proxy-sar-config\"" May 25 11:36:10.115668 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.115644 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:36:10.197463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.197428 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srgs\" (UniqueName: \"kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.197611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.197494 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.197611 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.197538 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.298457 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.298428 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.298616 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.298481 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7srgs\" (UniqueName: \"kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.298616 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.298512 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.298616 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:10.298607 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-412bf-predictor-serving-cert: secret "success-200-isvc-412bf-predictor-serving-cert" not found May 25 11:36:10.298815 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:10.298680 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls podName:836f3801-5cf4-468d-9998-99105397f7ab nodeName:}" failed. No retries permitted until 2026-05-25 11:36:10.798664666 +0000 UTC m=+1110.810273098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls") pod "success-200-isvc-412bf-predictor-556fc6f8-2s6tc" (UID: "836f3801-5cf4-468d-9998-99105397f7ab") : secret "success-200-isvc-412bf-predictor-serving-cert" not found May 25 11:36:10.299156 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.299135 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.328755 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.328734 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srgs\" (UniqueName: \"kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.462408 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.462387 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:36:10.500598 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.500578 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") pod \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " May 25 11:36:10.500709 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.500634 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle\") pod \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\" (UID: \"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457\") " May 25 11:36:10.501023 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.500993 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" (UID: "c93fcb61-4aaf-4d32-a09a-1f8a2cc18457"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:36:10.502701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.502679 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" (UID: "c93fcb61-4aaf-4d32-a09a-1f8a2cc18457"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:36:10.605641 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.605613 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:10.605641 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.605640 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:10.745984 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.745946 2548 generic.go:358] "Generic (PLEG): container finished" podID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerID="5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34" exitCode=2 May 25 11:36:10.746184 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.746018 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerDied","Data":"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34"} May 25 11:36:10.746965 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.746942 2548 generic.go:358] "Generic (PLEG): container finished" podID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerID="b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd" exitCode=0 May 25 11:36:10.747129 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.746981 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" event={"ID":"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457","Type":"ContainerDied","Data":"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd"} May 25 11:36:10.747129 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.747011 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" event={"ID":"c93fcb61-4aaf-4d32-a09a-1f8a2cc18457","Type":"ContainerDied","Data":"cdf2b2faf6f3ebeb9f8f42d60a21caab86faacd52525d8a6f039c1a57c0ad1b2"} May 25 11:36:10.747129 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.747031 2548 scope.go:117] "RemoveContainer" containerID="b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd" May 25 11:36:10.747129 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.747053 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767" May 25 11:36:10.754897 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.754883 2548 scope.go:117] "RemoveContainer" containerID="b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd" May 25 11:36:10.755166 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:10.755146 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd\": container with ID starting with b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd not found: ID does not exist" containerID="b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd" May 25 11:36:10.755259 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.755170 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd"} err="failed to get container status \"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd\": rpc error: code = NotFound desc = could not find container \"b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd\": container with ID starting with b267db52398c1c1c92b24c5ef505eab1a4c389ea1d35a207088215abbf11f8cd not found: ID does not exist" May 25 11:36:10.768530 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.768509 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:36:10.780073 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.780053 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-55329-668cc95c65-m2767"] May 25 11:36:10.806991 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.806972 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:10.809247 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:10.809230 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") pod \"success-200-isvc-412bf-predictor-556fc6f8-2s6tc\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:11.000814 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.000746 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:11.137864 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.137831 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:36:11.142008 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:36:11.141979 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f3801_5cf4_468d_9998_99105397f7ab.slice/crio-cca807c929e5f580aa9fdc247b8c9aae53f8a5e5bbfa1f0219a51613e08e506b WatchSource:0}: Error finding container cca807c929e5f580aa9fdc247b8c9aae53f8a5e5bbfa1f0219a51613e08e506b: Status 404 returned error can't find the container with id cca807c929e5f580aa9fdc247b8c9aae53f8a5e5bbfa1f0219a51613e08e506b May 25 11:36:11.751648 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.751616 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerStarted","Data":"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b"} May 25 11:36:11.751648 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.751650 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerStarted","Data":"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8"} May 25 11:36:11.751876 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.751661 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerStarted","Data":"cca807c929e5f580aa9fdc247b8c9aae53f8a5e5bbfa1f0219a51613e08e506b"} May 25 11:36:11.751876 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.751775 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:11.777378 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:11.777334 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podStartSLOduration=1.777319691 podStartE2EDuration="1.777319691s" podCreationTimestamp="2026-05-25 11:36:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:11.77598639 +0000 UTC m=+1111.787594846" watchObservedRunningTime="2026-05-25 11:36:11.777319691 +0000 UTC m=+1111.788928146" May 25 11:36:12.600151 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:12.600094 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" path="/var/lib/kubelet/pods/c93fcb61-4aaf-4d32-a09a-1f8a2cc18457/volumes" May 25 11:36:12.754724 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:12.754695 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:12.755905 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:12.755878 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:13.095422 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.095403 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:36:13.226802 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.226735 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " May 25 11:36:13.226802 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.226780 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpv4m\" (UniqueName: \"kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m\") pod \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " May 25 11:36:13.226972 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.226803 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") pod \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\" (UID: \"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97\") " May 25 11:36:13.227087 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.227054 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-f16f5-kube-rbac-proxy-sar-config") pod "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" (UID: "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97"). InnerVolumeSpecName "success-200-isvc-f16f5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:36:13.228957 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.228936 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" (UID: "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:36:13.229034 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.228985 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m" (OuterVolumeSpecName: "kube-api-access-jpv4m") pod "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" (UID: "d694fff0-c01e-41f2-8e8f-0fb8fb79cb97"). InnerVolumeSpecName "kube-api-access-jpv4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:36:13.328179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.328143 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:13.328179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.328178 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-success-200-isvc-f16f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:13.328179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.328188 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpv4m\" (UniqueName: \"kubernetes.io/projected/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97-kube-api-access-jpv4m\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:13.758766 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.758729 2548 generic.go:358] "Generic (PLEG): container finished" podID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerID="fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3" exitCode=0 May 25 11:36:13.759208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.758809 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" May 25 11:36:13.759208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.758811 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerDied","Data":"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3"} May 25 11:36:13.759208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.758845 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs" event={"ID":"d694fff0-c01e-41f2-8e8f-0fb8fb79cb97","Type":"ContainerDied","Data":"86f523dc160eaa299b0cf1253deba04b05aa9daa53f3a4be17a9c95e0303eacf"} May 25 11:36:13.759208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.758860 2548 scope.go:117] "RemoveContainer" containerID="5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34" May 25 11:36:13.759460 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.759437 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:13.766506 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.766489 2548 scope.go:117] "RemoveContainer" containerID="fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3" May 25 11:36:13.774931 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.774910 2548 scope.go:117] "RemoveContainer" containerID="5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34" May 25 11:36:13.775253 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:13.775226 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34\": container with ID starting with 5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34 not found: ID does not exist" containerID="5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34" May 25 11:36:13.775392 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.775262 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34"} err="failed to get container status \"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34\": rpc error: code = NotFound desc = could not find container \"5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34\": container with ID starting with 5eaaa08152355a95eb253da7978af7f45a24f0b4cf900d54f9e9355ddca3ac34 not found: ID does not exist" May 25 11:36:13.775392 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.775286 2548 scope.go:117] "RemoveContainer" containerID="fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3" May 25 11:36:13.775558 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:13.775537 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3\": container with ID starting with fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3 not found: ID does not exist" containerID="fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3" May 25 11:36:13.775606 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.775567 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3"} err="failed to get container status \"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3\": rpc error: code = NotFound desc = could not find container \"fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3\": container with ID starting with fae71272b80d943053cea8c389cee5fd81410ab3e79c085387be4475887b6ea3 not found: ID does not exist" May 25 11:36:13.789871 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.789850 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:36:13.808484 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:13.808460 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f16f5-predictor-db96fdd8d-nbpfs"] May 25 11:36:14.599243 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:14.599211 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" path="/var/lib/kubelet/pods/d694fff0-c01e-41f2-8e8f-0fb8fb79cb97/volumes" May 25 11:36:17.668199 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:17.668161 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 25 11:36:18.763155 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:18.763121 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:18.763549 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:18.763523 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:27.668800 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:27.668765 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:36:28.764049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:28.764002 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:38.763700 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:38.763655 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:40.637951 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.637917 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638270 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638287 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638301 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kube-rbac-proxy" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638309 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kube-rbac-proxy" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638332 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638341 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" May 25 11:36:40.638421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638413 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kserve-container" May 25 11:36:40.638742 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638427 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d694fff0-c01e-41f2-8e8f-0fb8fb79cb97" containerName="kube-rbac-proxy" May 25 11:36:40.638742 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.638441 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="c93fcb61-4aaf-4d32-a09a-1f8a2cc18457" containerName="switch-graph-55329" May 25 11:36:40.642559 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.642540 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.645239 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.645216 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-49543-serving-cert\"" May 25 11:36:40.645359 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.645238 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-49543-kube-rbac-proxy-sar-config\"" May 25 11:36:40.654333 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.654307 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:36:40.742093 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.742045 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.742093 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.742089 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.842693 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.842606 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.842693 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.842640 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.843314 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.843289 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.845274 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.845251 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls\") pod \"ensemble-graph-49543-6ccd8b5c87-lwpqk\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:40.956276 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:40.956238 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:41.083797 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:41.083765 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:36:41.086905 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:36:41.086878 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99536bae_c9b2_4d1d_865a_f10ed90b9222.slice/crio-69726ebaa0cc1f4c288c258ef3be421708aeab504154a60a037aaa4faa4d5e6b WatchSource:0}: Error finding container 69726ebaa0cc1f4c288c258ef3be421708aeab504154a60a037aaa4faa4d5e6b: Status 404 returned error can't find the container with id 69726ebaa0cc1f4c288c258ef3be421708aeab504154a60a037aaa4faa4d5e6b May 25 11:36:41.841250 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:41.841204 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" event={"ID":"99536bae-c9b2-4d1d-865a-f10ed90b9222","Type":"ContainerStarted","Data":"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262"} May 25 11:36:41.841250 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:41.841256 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" event={"ID":"99536bae-c9b2-4d1d-865a-f10ed90b9222","Type":"ContainerStarted","Data":"69726ebaa0cc1f4c288c258ef3be421708aeab504154a60a037aaa4faa4d5e6b"} May 25 11:36:41.841689 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:41.841344 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:41.858657 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:41.858613 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podStartSLOduration=1.858600163 podStartE2EDuration="1.858600163s" podCreationTimestamp="2026-05-25 11:36:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:41.857173894 +0000 UTC m=+1141.868782359" watchObservedRunningTime="2026-05-25 11:36:41.858600163 +0000 UTC m=+1141.870208674" May 25 11:36:47.850594 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:47.850563 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:36:48.763711 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:48.763632 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 25 11:36:50.702065 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.702027 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:36:50.702444 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.702285 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" containerID="cri-o://857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262" gracePeriod=30 May 25 11:36:50.799613 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.799577 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:36:50.799990 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.799943 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" containerID="cri-o://78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a" gracePeriod=30 May 25 11:36:50.799990 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.799972 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kube-rbac-proxy" containerID="cri-o://e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348" gracePeriod=30 May 25 11:36:50.848239 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.848213 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:36:50.852489 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.852473 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:50.855530 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.855509 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-20622-predictor-serving-cert\"" May 25 11:36:50.855973 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.855956 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-20622-kube-rbac-proxy-sar-config\"" May 25 11:36:50.869207 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:50.869171 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:36:51.013846 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.013783 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76vg\" (UniqueName: \"kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.013976 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.013859 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.013976 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.013896 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.114894 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.114868 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.115032 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.114916 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f76vg\" (UniqueName: \"kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.115032 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.114954 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.115581 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.115561 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.117448 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.117428 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.123610 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.123580 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76vg\" (UniqueName: \"kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg\") pod \"success-200-isvc-20622-predictor-68cdb5545b-mbbc8\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.163406 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.163383 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.288035 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.287968 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:36:51.290937 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:36:51.290905 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c8511_fc5f_4c2a_bc27_8f6370f73c3b.slice/crio-ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd WatchSource:0}: Error finding container ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd: Status 404 returned error can't find the container with id ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd May 25 11:36:51.870162 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.870131 2548 generic.go:358] "Generic (PLEG): container finished" podID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerID="e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348" exitCode=2 May 25 11:36:51.870536 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.870194 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerDied","Data":"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348"} May 25 11:36:51.871688 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.871664 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerStarted","Data":"0a652f956c2d60d0eef1e36b9e84c9d503a304a93eca7930cac7837010449a24"} May 25 11:36:51.871777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.871693 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerStarted","Data":"9a1dc42e0d8df870ad6d153d1c7e6a3dbc7b1530163cfef0e2b78f1f3d7a6c27"} May 25 11:36:51.871777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.871703 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerStarted","Data":"ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd"} May 25 11:36:51.871846 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.871835 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:51.892826 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:51.892785 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podStartSLOduration=1.892773585 podStartE2EDuration="1.892773585s" podCreationTimestamp="2026-05-25 11:36:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:51.891671495 +0000 UTC m=+1151.903279974" watchObservedRunningTime="2026-05-25 11:36:51.892773585 +0000 UTC m=+1151.904382040" May 25 11:36:52.663349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:52.663311 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused" May 25 11:36:52.848049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:52.848012 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:36:52.874572 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:52.874547 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:52.875734 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:52.875706 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:36:53.876799 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:53.876762 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:36:54.044461 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.044430 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:36:54.137648 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.137613 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config\") pod \"b2411ec6-a6aa-478a-8042-b43378fb3912\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " May 25 11:36:54.137792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.137665 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2rhv\" (UniqueName: \"kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv\") pod \"b2411ec6-a6aa-478a-8042-b43378fb3912\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " May 25 11:36:54.137792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.137712 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls\") pod \"b2411ec6-a6aa-478a-8042-b43378fb3912\" (UID: \"b2411ec6-a6aa-478a-8042-b43378fb3912\") " May 25 11:36:54.137982 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.137960 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-49543-kube-rbac-proxy-sar-config") pod "b2411ec6-a6aa-478a-8042-b43378fb3912" (UID: "b2411ec6-a6aa-478a-8042-b43378fb3912"). InnerVolumeSpecName "success-200-isvc-49543-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:36:54.139861 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.139827 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv" (OuterVolumeSpecName: "kube-api-access-c2rhv") pod "b2411ec6-a6aa-478a-8042-b43378fb3912" (UID: "b2411ec6-a6aa-478a-8042-b43378fb3912"). InnerVolumeSpecName "kube-api-access-c2rhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:36:54.139968 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.139910 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2411ec6-a6aa-478a-8042-b43378fb3912" (UID: "b2411ec6-a6aa-478a-8042-b43378fb3912"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:36:54.239257 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.239231 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2411ec6-a6aa-478a-8042-b43378fb3912-success-200-isvc-49543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:54.239346 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.239263 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rhv\" (UniqueName: \"kubernetes.io/projected/b2411ec6-a6aa-478a-8042-b43378fb3912-kube-api-access-c2rhv\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:54.239346 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.239280 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2411ec6-a6aa-478a-8042-b43378fb3912-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:36:54.884666 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.884595 2548 generic.go:358] "Generic (PLEG): container finished" podID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerID="78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a" exitCode=0 May 25 11:36:54.884666 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.884648 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerDied","Data":"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a"} May 25 11:36:54.885068 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.884679 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" event={"ID":"b2411ec6-a6aa-478a-8042-b43378fb3912","Type":"ContainerDied","Data":"931fa21500b7872fe065ab828cc7ba27d3ce80dd95e85254e3c63b8891a26c21"} May 25 11:36:54.885068 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.884697 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6" May 25 11:36:54.885068 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.884700 2548 scope.go:117] "RemoveContainer" containerID="e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348" May 25 11:36:54.892481 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.892462 2548 scope.go:117] "RemoveContainer" containerID="78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a" May 25 11:36:54.902564 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.902545 2548 scope.go:117] "RemoveContainer" containerID="e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348" May 25 11:36:54.902842 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:54.902821 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348\": container with ID starting with e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348 not found: ID does not exist" containerID="e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348" May 25 11:36:54.902977 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.902853 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348"} err="failed to get container status \"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348\": rpc error: code = NotFound desc = could not find container \"e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348\": container with ID starting with e556bd3932247ee1c4d13801c99d2f42db28c7397233982a6b13bd0e2c7fd348 not found: ID does not exist" May 25 11:36:54.902977 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.902878 2548 scope.go:117] "RemoveContainer" containerID="78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a" May 25 11:36:54.903177 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:36:54.903154 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a\": container with ID starting with 78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a not found: ID does not exist" containerID="78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a" May 25 11:36:54.903228 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.903186 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a"} err="failed to get container status \"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a\": rpc error: code = NotFound desc = could not find container \"78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a\": container with ID starting with 78301e2087325c3338f1387d4d8509a29712d947d9b2fb9e4fc0175fc49a324a not found: ID does not exist" May 25 11:36:54.903473 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.903461 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:36:54.904470 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:54.904450 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49543-predictor-75fc554798-2v6w6"] May 25 11:36:56.599218 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:56.599188 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" path="/var/lib/kubelet/pods/b2411ec6-a6aa-478a-8042-b43378fb3912/volumes" May 25 11:36:57.848673 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:57.848635 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:36:58.764808 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:58.764783 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:36:58.880551 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:58.880524 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:36:58.881125 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:36:58.881077 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:37:02.848322 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:02.848279 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:02.848691 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:02.848392 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:37:07.848049 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:07.848010 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:08.881769 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:08.881726 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:37:10.105937 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.105903 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:10.106360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106344 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kube-rbac-proxy" May 25 11:37:10.106411 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106363 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kube-rbac-proxy" May 25 11:37:10.106411 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106383 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" May 25 11:37:10.106411 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106392 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" May 25 11:37:10.106503 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106452 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kserve-container" May 25 11:37:10.106503 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.106465 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2411ec6-a6aa-478a-8042-b43378fb3912" containerName="kube-rbac-proxy" May 25 11:37:10.110969 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.110952 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.113129 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.113082 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-412bf-serving-cert\"" May 25 11:37:10.113221 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.113126 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-412bf-kube-rbac-proxy-sar-config\"" May 25 11:37:10.117685 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.117666 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:10.149997 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.149969 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.150092 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.150016 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.251011 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.250987 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.251149 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.251021 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.251149 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:10.251135 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-412bf-serving-cert: secret "sequence-graph-412bf-serving-cert" not found May 25 11:37:10.251263 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:10.251187 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls podName:524390a7-82fd-4c7e-991d-6298fbe2b538 nodeName:}" failed. No retries permitted until 2026-05-25 11:37:10.751172957 +0000 UTC m=+1170.762781395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls") pod "sequence-graph-412bf-5fd95869d7-vz7sb" (UID: "524390a7-82fd-4c7e-991d-6298fbe2b538") : secret "sequence-graph-412bf-serving-cert" not found May 25 11:37:10.251580 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.251562 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.754915 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.754888 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:10.757373 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:10.757352 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") pod \"sequence-graph-412bf-5fd95869d7-vz7sb\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:11.022281 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.022225 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:11.143135 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.143081 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:11.145281 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:37:11.145253 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524390a7_82fd_4c7e_991d_6298fbe2b538.slice/crio-8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89 WatchSource:0}: Error finding container 8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89: Status 404 returned error can't find the container with id 8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89 May 25 11:37:11.931613 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.931575 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" event={"ID":"524390a7-82fd-4c7e-991d-6298fbe2b538","Type":"ContainerStarted","Data":"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313"} May 25 11:37:11.931613 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.931615 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" event={"ID":"524390a7-82fd-4c7e-991d-6298fbe2b538","Type":"ContainerStarted","Data":"8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89"} May 25 11:37:11.931815 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.931648 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:11.949426 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:11.949386 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podStartSLOduration=1.949372759 podStartE2EDuration="1.949372759s" podCreationTimestamp="2026-05-25 11:37:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:11.947881675 +0000 UTC m=+1171.959490130" watchObservedRunningTime="2026-05-25 11:37:11.949372759 +0000 UTC m=+1171.960981213" May 25 11:37:12.847969 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:12.847929 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:17.847947 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:17.847910 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:17.939761 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:17.939735 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:18.881683 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:18.881643 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:37:20.170724 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.170694 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:20.171139 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.170901 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" containerID="cri-o://05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313" gracePeriod=30 May 25 11:37:20.292359 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.292326 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:37:20.292695 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.292668 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" containerID="cri-o://015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8" gracePeriod=30 May 25 11:37:20.292782 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.292720 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kube-rbac-proxy" containerID="cri-o://e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b" gracePeriod=30 May 25 11:37:20.328042 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.328019 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:37:20.331407 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.331390 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.336592 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.336541 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\"" May 25 11:37:20.337075 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.337056 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b59e4-predictor-serving-cert\"" May 25 11:37:20.349087 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.349065 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:37:20.420477 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.420431 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpt8\" (UniqueName: \"kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.420602 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.420480 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.420602 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.420577 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.521744 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.521684 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.521744 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.521735 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpt8\" (UniqueName: \"kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.521949 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.521766 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.521949 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:20.521829 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-serving-cert: secret "success-200-isvc-b59e4-predictor-serving-cert" not found May 25 11:37:20.521949 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:20.521897 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls podName:1c459914-3782-4a1d-a188-2a60021bb906 nodeName:}" failed. No retries permitted until 2026-05-25 11:37:21.021876464 +0000 UTC m=+1181.033484900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls") pod "success-200-isvc-b59e4-predictor-7dfbdf574-768rn" (UID: "1c459914-3782-4a1d-a188-2a60021bb906") : secret "success-200-isvc-b59e4-predictor-serving-cert" not found May 25 11:37:20.522371 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.522353 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.533285 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.533262 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpt8\" (UniqueName: \"kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:20.885219 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.885196 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:37:20.925689 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.925648 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle\") pod \"99536bae-c9b2-4d1d-865a-f10ed90b9222\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " May 25 11:37:20.925827 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.925706 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls\") pod \"99536bae-c9b2-4d1d-865a-f10ed90b9222\" (UID: \"99536bae-c9b2-4d1d-865a-f10ed90b9222\") " May 25 11:37:20.925943 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.925919 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "99536bae-c9b2-4d1d-865a-f10ed90b9222" (UID: "99536bae-c9b2-4d1d-865a-f10ed90b9222"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:37:20.927699 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.927679 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99536bae-c9b2-4d1d-865a-f10ed90b9222" (UID: "99536bae-c9b2-4d1d-865a-f10ed90b9222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:37:20.955160 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.955137 2548 generic.go:358] "Generic (PLEG): container finished" podID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerID="857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262" exitCode=0 May 25 11:37:20.955273 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.955196 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" May 25 11:37:20.955273 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.955213 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" event={"ID":"99536bae-c9b2-4d1d-865a-f10ed90b9222","Type":"ContainerDied","Data":"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262"} May 25 11:37:20.955273 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.955249 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk" event={"ID":"99536bae-c9b2-4d1d-865a-f10ed90b9222","Type":"ContainerDied","Data":"69726ebaa0cc1f4c288c258ef3be421708aeab504154a60a037aaa4faa4d5e6b"} May 25 11:37:20.955273 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.955269 2548 scope.go:117] "RemoveContainer" containerID="857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262" May 25 11:37:20.956805 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.956770 2548 generic.go:358] "Generic (PLEG): container finished" podID="836f3801-5cf4-468d-9998-99105397f7ab" containerID="e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b" exitCode=2 May 25 11:37:20.956894 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.956814 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerDied","Data":"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b"} May 25 11:37:20.963766 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.963750 2548 scope.go:117] "RemoveContainer" containerID="857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262" May 25 11:37:20.964200 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:20.964148 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262\": container with ID starting with 857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262 not found: ID does not exist" containerID="857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262" May 25 11:37:20.964200 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.964182 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262"} err="failed to get container status \"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262\": rpc error: code = NotFound desc = could not find container \"857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262\": container with ID starting with 857737f7cd757bd8485492713dd6b69f5414719d03c5f1ba730e302c4faae262 not found: ID does not exist" May 25 11:37:20.982841 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.982818 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:37:20.986708 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:20.986687 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-49543-6ccd8b5c87-lwpqk"] May 25 11:37:21.026937 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.026914 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:21.027043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.026993 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99536bae-c9b2-4d1d-865a-f10ed90b9222-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:21.027043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.027012 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99536bae-c9b2-4d1d-865a-f10ed90b9222-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:21.029306 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.029289 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") pod \"success-200-isvc-b59e4-predictor-7dfbdf574-768rn\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:21.241463 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.241427 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:21.367798 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.367765 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:37:21.370902 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:37:21.370871 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c459914_3782_4a1d_a188_2a60021bb906.slice/crio-44799772dbe20a58cd7c4f7f4b08452d6d1e5b6aef3e5792a1d2ea9fefa55e2b WatchSource:0}: Error finding container 44799772dbe20a58cd7c4f7f4b08452d6d1e5b6aef3e5792a1d2ea9fefa55e2b: Status 404 returned error can't find the container with id 44799772dbe20a58cd7c4f7f4b08452d6d1e5b6aef3e5792a1d2ea9fefa55e2b May 25 11:37:21.961607 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.961563 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerStarted","Data":"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41"} May 25 11:37:21.961607 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.961613 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerStarted","Data":"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958"} May 25 11:37:21.961807 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.961629 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerStarted","Data":"44799772dbe20a58cd7c4f7f4b08452d6d1e5b6aef3e5792a1d2ea9fefa55e2b"} May 25 11:37:21.980806 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:21.980765 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podStartSLOduration=1.980752678 podStartE2EDuration="1.980752678s" podCreationTimestamp="2026-05-25 11:37:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:21.980077037 +0000 UTC m=+1181.991685493" watchObservedRunningTime="2026-05-25 11:37:21.980752678 +0000 UTC m=+1181.992361133" May 25 11:37:22.600774 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:22.600734 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" path="/var/lib/kubelet/pods/99536bae-c9b2-4d1d-865a-f10ed90b9222/volumes" May 25 11:37:22.938695 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:22.938654 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:22.965084 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:22.965058 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:22.965201 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:22.965093 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:22.966293 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:22.966271 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:37:23.133843 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.133811 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:37:23.247713 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.247649 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7srgs\" (UniqueName: \"kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs\") pod \"836f3801-5cf4-468d-9998-99105397f7ab\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " May 25 11:37:23.247713 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.247711 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") pod \"836f3801-5cf4-468d-9998-99105397f7ab\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " May 25 11:37:23.247856 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.247733 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"836f3801-5cf4-468d-9998-99105397f7ab\" (UID: \"836f3801-5cf4-468d-9998-99105397f7ab\") " May 25 11:37:23.248176 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.248150 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-412bf-kube-rbac-proxy-sar-config") pod "836f3801-5cf4-468d-9998-99105397f7ab" (UID: "836f3801-5cf4-468d-9998-99105397f7ab"). InnerVolumeSpecName "success-200-isvc-412bf-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:37:23.249892 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.249868 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs" (OuterVolumeSpecName: "kube-api-access-7srgs") pod "836f3801-5cf4-468d-9998-99105397f7ab" (UID: "836f3801-5cf4-468d-9998-99105397f7ab"). InnerVolumeSpecName "kube-api-access-7srgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:37:23.249892 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.249878 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "836f3801-5cf4-468d-9998-99105397f7ab" (UID: "836f3801-5cf4-468d-9998-99105397f7ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:37:23.348360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.348336 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srgs\" (UniqueName: \"kubernetes.io/projected/836f3801-5cf4-468d-9998-99105397f7ab-kube-api-access-7srgs\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:23.348360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.348362 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836f3801-5cf4-468d-9998-99105397f7ab-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:23.348483 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.348372 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836f3801-5cf4-468d-9998-99105397f7ab-success-200-isvc-412bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:23.968756 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.968723 2548 generic.go:358] "Generic (PLEG): container finished" podID="836f3801-5cf4-468d-9998-99105397f7ab" containerID="015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8" exitCode=0 May 25 11:37:23.969166 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.968798 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" May 25 11:37:23.969166 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.968815 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerDied","Data":"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8"} May 25 11:37:23.969166 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.968856 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc" event={"ID":"836f3801-5cf4-468d-9998-99105397f7ab","Type":"ContainerDied","Data":"cca807c929e5f580aa9fdc247b8c9aae53f8a5e5bbfa1f0219a51613e08e506b"} May 25 11:37:23.969166 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.968874 2548 scope.go:117] "RemoveContainer" containerID="e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b" May 25 11:37:23.969453 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.969424 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:37:23.977349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.977332 2548 scope.go:117] "RemoveContainer" containerID="015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8" May 25 11:37:23.984134 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.984116 2548 scope.go:117] "RemoveContainer" containerID="e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b" May 25 11:37:23.984385 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:23.984367 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b\": container with ID starting with e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b not found: ID does not exist" containerID="e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b" May 25 11:37:23.984430 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.984393 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b"} err="failed to get container status \"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b\": rpc error: code = NotFound desc = could not find container \"e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b\": container with ID starting with e15487d9a9183abe507eeab1d9e8cd0e54d806d236f49fcbe2b507027dccbd7b not found: ID does not exist" May 25 11:37:23.984430 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.984408 2548 scope.go:117] "RemoveContainer" containerID="015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8" May 25 11:37:23.984654 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:23.984635 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8\": container with ID starting with 015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8 not found: ID does not exist" containerID="015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8" May 25 11:37:23.984710 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.984664 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8"} err="failed to get container status \"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8\": rpc error: code = NotFound desc = could not find container \"015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8\": container with ID starting with 015259a265965b33fb6c79e0c15ad65ab5f2c5febbc112343ccf231caf26def8 not found: ID does not exist" May 25 11:37:23.990347 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.990325 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:37:23.992899 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:23.992883 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-412bf-predictor-556fc6f8-2s6tc"] May 25 11:37:24.599269 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:24.599240 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836f3801-5cf4-468d-9998-99105397f7ab" path="/var/lib/kubelet/pods/836f3801-5cf4-468d-9998-99105397f7ab/volumes" May 25 11:37:27.938373 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:27.938328 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:28.881462 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:28.881426 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 25 11:37:28.974356 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:28.974329 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:37:28.974867 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:28.974834 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:37:32.938885 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:32.938842 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:32.939329 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:32.938968 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:37.938236 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:37.938200 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:38.882465 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:38.882437 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:37:38.974838 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:38.974806 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:37:40.557429 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:40.557401 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:37:40.559660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:40.559640 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:37:42.938740 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:42.938694 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:47.939563 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:47.939509 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:37:48.975257 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:48.975214 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:37:50.226944 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:50.226920 2548 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524390a7_82fd_4c7e_991d_6298fbe2b538.slice/crio-8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89\": RecentStats: unable to find data in memory cache]" May 25 11:37:50.347996 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.347972 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:50.434370 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.434297 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle\") pod \"524390a7-82fd-4c7e-991d-6298fbe2b538\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " May 25 11:37:50.434370 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.434351 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") pod \"524390a7-82fd-4c7e-991d-6298fbe2b538\" (UID: \"524390a7-82fd-4c7e-991d-6298fbe2b538\") " May 25 11:37:50.434637 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.434615 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "524390a7-82fd-4c7e-991d-6298fbe2b538" (UID: "524390a7-82fd-4c7e-991d-6298fbe2b538"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:37:50.436657 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.436627 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "524390a7-82fd-4c7e-991d-6298fbe2b538" (UID: "524390a7-82fd-4c7e-991d-6298fbe2b538"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:37:50.534845 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.534820 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/524390a7-82fd-4c7e-991d-6298fbe2b538-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:50.534929 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.534854 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/524390a7-82fd-4c7e-991d-6298fbe2b538-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:37:50.942870 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.942840 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:37:50.943147 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943093 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kube-rbac-proxy" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943150 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kube-rbac-proxy" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943161 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943166 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943178 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943184 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943191 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" May 25 11:37:50.943198 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943197 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" May 25 11:37:50.943399 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943240 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerName="sequence-graph-412bf" May 25 11:37:50.943399 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943249 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kserve-container" May 25 11:37:50.943399 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943255 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="836f3801-5cf4-468d-9998-99105397f7ab" containerName="kube-rbac-proxy" May 25 11:37:50.943399 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.943262 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="99536bae-c9b2-4d1d-865a-f10ed90b9222" containerName="ensemble-graph-49543" May 25 11:37:50.946195 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.946178 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:50.948477 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.948457 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-20622-serving-cert\"" May 25 11:37:50.948587 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.948522 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-20622-kube-rbac-proxy-sar-config\"" May 25 11:37:50.954983 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:50.954964 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:37:51.039026 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.039002 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.039167 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.039043 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.042051 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.042025 2548 generic.go:358] "Generic (PLEG): container finished" podID="524390a7-82fd-4c7e-991d-6298fbe2b538" containerID="05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313" exitCode=0 May 25 11:37:51.042159 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.042085 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" May 25 11:37:51.042159 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.042087 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" event={"ID":"524390a7-82fd-4c7e-991d-6298fbe2b538","Type":"ContainerDied","Data":"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313"} May 25 11:37:51.042159 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.042140 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb" event={"ID":"524390a7-82fd-4c7e-991d-6298fbe2b538","Type":"ContainerDied","Data":"8c4218a5b31de0b1c9421415aa03a571d097ea5a72f49d1c60075fbce5afef89"} May 25 11:37:51.042159 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.042158 2548 scope.go:117] "RemoveContainer" containerID="05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313" May 25 11:37:51.049780 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.049763 2548 scope.go:117] "RemoveContainer" containerID="05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313" May 25 11:37:51.050042 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:51.050024 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313\": container with ID starting with 05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313 not found: ID does not exist" containerID="05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313" May 25 11:37:51.050088 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.050051 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313"} err="failed to get container status \"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313\": rpc error: code = NotFound desc = could not find container \"05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313\": container with ID starting with 05172b1de19249a3971dd81a3659992e7ea2dff5a1bd4dcdbd9e8df8aa432313 not found: ID does not exist" May 25 11:37:51.057970 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.057944 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:51.059387 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.059369 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-412bf-5fd95869d7-vz7sb"] May 25 11:37:51.140011 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.139990 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.140087 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.140024 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.140161 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:51.140140 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-20622-serving-cert: secret "ensemble-graph-20622-serving-cert" not found May 25 11:37:51.140210 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:37:51.140200 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls podName:d328ad6c-d586-4706-8021-e4876cd7c03c nodeName:}" failed. No retries permitted until 2026-05-25 11:37:51.640183607 +0000 UTC m=+1211.651792045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls") pod "ensemble-graph-20622-549955698d-5qphk" (UID: "d328ad6c-d586-4706-8021-e4876cd7c03c") : secret "ensemble-graph-20622-serving-cert" not found May 25 11:37:51.140585 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.140569 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.643709 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.643680 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.646031 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.646011 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") pod \"ensemble-graph-20622-549955698d-5qphk\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.856290 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.856264 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:51.980215 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:51.980188 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:37:51.982324 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:37:51.982296 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd328ad6c_d586_4706_8021_e4876cd7c03c.slice/crio-131aeef375e9beb74ca40f13d93aca77b0ab32f70e18cbd8faf4a6a0b0280a47 WatchSource:0}: Error finding container 131aeef375e9beb74ca40f13d93aca77b0ab32f70e18cbd8faf4a6a0b0280a47: Status 404 returned error can't find the container with id 131aeef375e9beb74ca40f13d93aca77b0ab32f70e18cbd8faf4a6a0b0280a47 May 25 11:37:52.051307 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:52.051272 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" event={"ID":"d328ad6c-d586-4706-8021-e4876cd7c03c","Type":"ContainerStarted","Data":"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689"} May 25 11:37:52.051442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:52.051315 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" event={"ID":"d328ad6c-d586-4706-8021-e4876cd7c03c","Type":"ContainerStarted","Data":"131aeef375e9beb74ca40f13d93aca77b0ab32f70e18cbd8faf4a6a0b0280a47"} May 25 11:37:52.051442 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:52.051410 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:52.069367 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:52.069327 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podStartSLOduration=2.06931324 podStartE2EDuration="2.06931324s" podCreationTimestamp="2026-05-25 11:37:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:52.06849121 +0000 UTC m=+1212.080099664" watchObservedRunningTime="2026-05-25 11:37:52.06931324 +0000 UTC m=+1212.080921685" May 25 11:37:52.599479 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:52.599450 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="524390a7-82fd-4c7e-991d-6298fbe2b538" path="/var/lib/kubelet/pods/524390a7-82fd-4c7e-991d-6298fbe2b538/volumes" May 25 11:37:58.060035 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:58.060008 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:37:58.975328 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:37:58.975295 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 25 11:38:08.975433 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:38:08.975403 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:42:40.575404 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:42:40.575380 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:42:40.578570 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:42:40.578552 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:46:05.626126 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.626035 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:46:05.626585 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.626280 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" containerID="cri-o://3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689" gracePeriod=30 May 25 11:46:05.753156 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.753124 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:46:05.753506 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.753458 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" containerID="cri-o://9a1dc42e0d8df870ad6d153d1c7e6a3dbc7b1530163cfef0e2b78f1f3d7a6c27" gracePeriod=30 May 25 11:46:05.753617 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.753493 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kube-rbac-proxy" containerID="cri-o://0a652f956c2d60d0eef1e36b9e84c9d503a304a93eca7930cac7837010449a24" gracePeriod=30 May 25 11:46:05.848265 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.848237 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:46:05.851456 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.851441 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.855204 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.855183 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3e68-predictor-serving-cert\"" May 25 11:46:05.855660 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.855580 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\"" May 25 11:46:05.875453 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.875415 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:46:05.884179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.884115 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.884179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.884160 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.884303 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.884227 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgsdc\" (UniqueName: \"kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.985557 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.985521 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zgsdc\" (UniqueName: \"kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.985672 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.985586 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.985672 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.985618 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.985758 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:46:05.985723 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-serving-cert: secret "success-200-isvc-c3e68-predictor-serving-cert" not found May 25 11:46:05.985793 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:46:05.985785 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls podName:ef0629fb-809f-47ab-8312-7f97a370c16b nodeName:}" failed. No retries permitted until 2026-05-25 11:46:06.485766135 +0000 UTC m=+1706.497374578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls") pod "success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" (UID: "ef0629fb-809f-47ab-8312-7f97a370c16b") : secret "success-200-isvc-c3e68-predictor-serving-cert" not found May 25 11:46:05.986362 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.986341 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:05.997522 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:05.997504 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgsdc\" (UniqueName: \"kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:06.476540 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.476507 2548 generic.go:358] "Generic (PLEG): container finished" podID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerID="0a652f956c2d60d0eef1e36b9e84c9d503a304a93eca7930cac7837010449a24" exitCode=2 May 25 11:46:06.476692 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.476558 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerDied","Data":"0a652f956c2d60d0eef1e36b9e84c9d503a304a93eca7930cac7837010449a24"} May 25 11:46:06.489811 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.489788 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:06.492264 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.492243 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") pod \"success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:06.763588 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.763514 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:06.890577 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.890541 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:46:06.895458 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:46:06.895429 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0629fb_809f_47ab_8312_7f97a370c16b.slice/crio-716e8bc5914ca8935960b1c682d8b5977b47ee2b609d2cdcedc9eb9021c52844 WatchSource:0}: Error finding container 716e8bc5914ca8935960b1c682d8b5977b47ee2b609d2cdcedc9eb9021c52844: Status 404 returned error can't find the container with id 716e8bc5914ca8935960b1c682d8b5977b47ee2b609d2cdcedc9eb9021c52844 May 25 11:46:06.897396 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:06.897378 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 11:46:07.481133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:07.481078 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerStarted","Data":"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b"} May 25 11:46:07.481133 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:07.481131 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerStarted","Data":"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8"} May 25 11:46:07.481327 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:07.481142 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerStarted","Data":"716e8bc5914ca8935960b1c682d8b5977b47ee2b609d2cdcedc9eb9021c52844"} May 25 11:46:07.481327 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:07.481231 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:07.502485 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:07.502434 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podStartSLOduration=2.502415688 podStartE2EDuration="2.502415688s" podCreationTimestamp="2026-05-25 11:46:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:46:07.501183282 +0000 UTC m=+1707.512791750" watchObservedRunningTime="2026-05-25 11:46:07.502415688 +0000 UTC m=+1707.514024145" May 25 11:46:08.058701 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.058662 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:08.485313 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.485285 2548 generic.go:358] "Generic (PLEG): container finished" podID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerID="9a1dc42e0d8df870ad6d153d1c7e6a3dbc7b1530163cfef0e2b78f1f3d7a6c27" exitCode=0 May 25 11:46:08.485443 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.485357 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerDied","Data":"9a1dc42e0d8df870ad6d153d1c7e6a3dbc7b1530163cfef0e2b78f1f3d7a6c27"} May 25 11:46:08.485443 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.485398 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" event={"ID":"379c8511-fc5f-4c2a-bc27-8f6370f73c3b","Type":"ContainerDied","Data":"ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd"} May 25 11:46:08.485443 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.485411 2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc0bb4c7baa8975a08d2e60da731c1442e3b202ca5ae30490f7baa9244625fd" May 25 11:46:08.485648 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.485636 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:08.486893 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.486871 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:08.492179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.492162 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:46:08.605563 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.605544 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls\") pod \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " May 25 11:46:08.605689 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.605628 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config\") pod \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " May 25 11:46:08.605689 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.605681 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f76vg\" (UniqueName: \"kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg\") pod \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\" (UID: \"379c8511-fc5f-4c2a-bc27-8f6370f73c3b\") " May 25 11:46:08.605979 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.605948 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-20622-kube-rbac-proxy-sar-config") pod "379c8511-fc5f-4c2a-bc27-8f6370f73c3b" (UID: "379c8511-fc5f-4c2a-bc27-8f6370f73c3b"). InnerVolumeSpecName "success-200-isvc-20622-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:46:08.607810 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.607765 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "379c8511-fc5f-4c2a-bc27-8f6370f73c3b" (UID: "379c8511-fc5f-4c2a-bc27-8f6370f73c3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:46:08.607989 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.607972 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg" (OuterVolumeSpecName: "kube-api-access-f76vg") pod "379c8511-fc5f-4c2a-bc27-8f6370f73c3b" (UID: "379c8511-fc5f-4c2a-bc27-8f6370f73c3b"). InnerVolumeSpecName "kube-api-access-f76vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:46:08.706336 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.706315 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-success-200-isvc-20622-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:08.706336 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.706338 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f76vg\" (UniqueName: \"kubernetes.io/projected/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-kube-api-access-f76vg\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:08.706460 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:08.706349 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379c8511-fc5f-4c2a-bc27-8f6370f73c3b-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:09.488318 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:09.488284 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8" May 25 11:46:09.488756 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:09.488593 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:09.509500 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:09.509471 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:46:09.513057 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:09.513036 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-20622-predictor-68cdb5545b-mbbc8"] May 25 11:46:10.598417 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:10.598384 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" path="/var/lib/kubelet/pods/379c8511-fc5f-4c2a-bc27-8f6370f73c3b/volumes" May 25 11:46:13.057953 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:13.057916 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:14.493411 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:14.493382 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:46:14.493988 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:14.493961 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:18.058484 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:18.058449 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:18.058864 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:18.058552 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:46:23.058191 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:23.058154 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:24.494814 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:24.494775 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:28.058309 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:28.058269 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:33.058775 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:33.058728 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:46:34.494911 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:34.494870 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:35.104143 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.104091 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:46:35.104502 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.104445 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" containerID="cri-o://c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958" gracePeriod=30 May 25 11:46:35.104634 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.104510 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kube-rbac-proxy" containerID="cri-o://0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41" gracePeriod=30 May 25 11:46:35.134895 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.134868 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 11:46:35.135172 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135159 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kube-rbac-proxy" May 25 11:46:35.135172 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135173 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kube-rbac-proxy" May 25 11:46:35.135281 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135191 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" May 25 11:46:35.135281 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135196 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" May 25 11:46:35.135281 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135241 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kserve-container" May 25 11:46:35.135281 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.135250 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="379c8511-fc5f-4c2a-bc27-8f6370f73c3b" containerName="kube-rbac-proxy" May 25 11:46:35.138297 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.138280 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.140725 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.140702 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1cab-predictor-serving-cert\"" May 25 11:46:35.140853 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.140757 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\"" May 25 11:46:35.149613 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.149591 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 11:46:35.285635 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.285598 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.285777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.285671 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8svk\" (UniqueName: \"kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.285777 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.285707 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.386271 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.386240 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l8svk\" (UniqueName: \"kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.386406 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.386281 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.386406 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.386324 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.387003 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.386942 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.388836 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.388813 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.394919 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.394894 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8svk\" (UniqueName: \"kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk\") pod \"success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.448538 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.448511 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:35.561271 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.561240 2548 generic.go:358] "Generic (PLEG): container finished" podID="1c459914-3782-4a1d-a188-2a60021bb906" containerID="0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41" exitCode=2 May 25 11:46:35.561604 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.561299 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerDied","Data":"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41"} May 25 11:46:35.581558 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.581535 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 11:46:35.583487 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:46:35.583461 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d5c0a0_73a1_4d64_84ec_54ef72df2661.slice/crio-0b669b70479712fd84ea357abca2d48388679557ed7f52a700f8c784d65b1003 WatchSource:0}: Error finding container 0b669b70479712fd84ea357abca2d48388679557ed7f52a700f8c784d65b1003: Status 404 returned error can't find the container with id 0b669b70479712fd84ea357abca2d48388679557ed7f52a700f8c784d65b1003 May 25 11:46:35.759743 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.759715 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:46:35.890810 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.890745 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") pod \"d328ad6c-d586-4706-8021-e4876cd7c03c\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " May 25 11:46:35.890810 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.890792 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle\") pod \"d328ad6c-d586-4706-8021-e4876cd7c03c\" (UID: \"d328ad6c-d586-4706-8021-e4876cd7c03c\") " May 25 11:46:35.891216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.891191 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d328ad6c-d586-4706-8021-e4876cd7c03c" (UID: "d328ad6c-d586-4706-8021-e4876cd7c03c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:46:35.892864 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.892844 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d328ad6c-d586-4706-8021-e4876cd7c03c" (UID: "d328ad6c-d586-4706-8021-e4876cd7c03c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:46:35.991809 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.991781 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d328ad6c-d586-4706-8021-e4876cd7c03c-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:35.991809 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:35.991806 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d328ad6c-d586-4706-8021-e4876cd7c03c-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:36.565584 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.565541 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerStarted","Data":"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca"} May 25 11:46:36.565584 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.565582 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerStarted","Data":"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1"} May 25 11:46:36.566190 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.565594 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerStarted","Data":"0b669b70479712fd84ea357abca2d48388679557ed7f52a700f8c784d65b1003"} May 25 11:46:36.566190 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.565669 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:36.566586 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.566565 2548 generic.go:358] "Generic (PLEG): container finished" podID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerID="3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689" exitCode=137 May 25 11:46:36.566668 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.566621 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" May 25 11:46:36.566711 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.566622 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" event={"ID":"d328ad6c-d586-4706-8021-e4876cd7c03c","Type":"ContainerDied","Data":"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689"} May 25 11:46:36.566753 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.566725 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk" event={"ID":"d328ad6c-d586-4706-8021-e4876cd7c03c","Type":"ContainerDied","Data":"131aeef375e9beb74ca40f13d93aca77b0ab32f70e18cbd8faf4a6a0b0280a47"} May 25 11:46:36.566753 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.566748 2548 scope.go:117] "RemoveContainer" containerID="3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689" May 25 11:46:36.574624 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.574608 2548 scope.go:117] "RemoveContainer" containerID="3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689" May 25 11:46:36.574864 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:46:36.574848 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689\": container with ID starting with 3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689 not found: ID does not exist" containerID="3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689" May 25 11:46:36.574922 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.574876 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689"} err="failed to get container status \"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689\": rpc error: code = NotFound desc = could not find container \"3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689\": container with ID starting with 3a2aadcd56e4b06839647ed77c2e1650be430e165f66589332bcbe9553cb3689 not found: ID does not exist" May 25 11:46:36.591782 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.591746 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podStartSLOduration=1.5917363789999999 podStartE2EDuration="1.591736379s" podCreationTimestamp="2026-05-25 11:46:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:46:36.586722181 +0000 UTC m=+1736.598330636" watchObservedRunningTime="2026-05-25 11:46:36.591736379 +0000 UTC m=+1736.603344834" May 25 11:46:36.606194 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.606172 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:46:36.610675 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:36.610658 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-20622-549955698d-5qphk"] May 25 11:46:37.570722 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:37.570694 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:37.571724 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:37.571699 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:46:38.057335 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.057312 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:46:38.206310 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.206287 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqpt8\" (UniqueName: \"kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8\") pod \"1c459914-3782-4a1d-a188-2a60021bb906\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " May 25 11:46:38.206439 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.206317 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") pod \"1c459914-3782-4a1d-a188-2a60021bb906\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " May 25 11:46:38.206439 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.206353 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"1c459914-3782-4a1d-a188-2a60021bb906\" (UID: \"1c459914-3782-4a1d-a188-2a60021bb906\") " May 25 11:46:38.206662 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.206640 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-b59e4-kube-rbac-proxy-sar-config") pod "1c459914-3782-4a1d-a188-2a60021bb906" (UID: "1c459914-3782-4a1d-a188-2a60021bb906"). InnerVolumeSpecName "success-200-isvc-b59e4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:46:38.208543 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.208521 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8" (OuterVolumeSpecName: "kube-api-access-pqpt8") pod "1c459914-3782-4a1d-a188-2a60021bb906" (UID: "1c459914-3782-4a1d-a188-2a60021bb906"). InnerVolumeSpecName "kube-api-access-pqpt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:46:38.208543 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.208530 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c459914-3782-4a1d-a188-2a60021bb906" (UID: "1c459914-3782-4a1d-a188-2a60021bb906"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:46:38.306980 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.306955 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqpt8\" (UniqueName: \"kubernetes.io/projected/1c459914-3782-4a1d-a188-2a60021bb906-kube-api-access-pqpt8\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:38.306980 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.306979 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c459914-3782-4a1d-a188-2a60021bb906-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:38.307160 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.306990 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c459914-3782-4a1d-a188-2a60021bb906-success-200-isvc-b59e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:46:38.574318 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574239 2548 generic.go:358] "Generic (PLEG): container finished" podID="1c459914-3782-4a1d-a188-2a60021bb906" containerID="c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958" exitCode=0 May 25 11:46:38.574738 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574319 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" May 25 11:46:38.574738 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574328 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerDied","Data":"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958"} May 25 11:46:38.574738 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574372 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn" event={"ID":"1c459914-3782-4a1d-a188-2a60021bb906","Type":"ContainerDied","Data":"44799772dbe20a58cd7c4f7f4b08452d6d1e5b6aef3e5792a1d2ea9fefa55e2b"} May 25 11:46:38.574738 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574395 2548 scope.go:117] "RemoveContainer" containerID="0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41" May 25 11:46:38.574947 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.574869 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:46:38.586866 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.586848 2548 scope.go:117] "RemoveContainer" containerID="c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958" May 25 11:46:38.593762 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.593741 2548 scope.go:117] "RemoveContainer" containerID="0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41" May 25 11:46:38.593999 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:46:38.593979 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41\": container with ID starting with 0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41 not found: ID does not exist" containerID="0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41" May 25 11:46:38.594058 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.594006 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41"} err="failed to get container status \"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41\": rpc error: code = NotFound desc = could not find container \"0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41\": container with ID starting with 0815d1c0de4df63c057d57e6fb22d20d7aeef726568e30dd666a019d28203d41 not found: ID does not exist" May 25 11:46:38.594058 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.594024 2548 scope.go:117] "RemoveContainer" containerID="c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958" May 25 11:46:38.594356 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:46:38.594339 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958\": container with ID starting with c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958 not found: ID does not exist" containerID="c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958" May 25 11:46:38.594414 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.594357 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958"} err="failed to get container status \"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958\": rpc error: code = NotFound desc = could not find container \"c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958\": container with ID starting with c9b53b2156f4040320debebf0f14e67963281cd6d0b596308f6571596381a958 not found: ID does not exist" May 25 11:46:38.598167 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.598144 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" path="/var/lib/kubelet/pods/d328ad6c-d586-4706-8021-e4876cd7c03c/volumes" May 25 11:46:38.599253 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.599231 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:46:38.603671 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:38.603650 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b59e4-predictor-7dfbdf574-768rn"] May 25 11:46:40.598461 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:40.598430 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c459914-3782-4a1d-a188-2a60021bb906" path="/var/lib/kubelet/pods/1c459914-3782-4a1d-a188-2a60021bb906/volumes" May 25 11:46:40.629954 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:40.629927 2548 scope.go:117] "RemoveContainer" containerID="0a652f956c2d60d0eef1e36b9e84c9d503a304a93eca7930cac7837010449a24" May 25 11:46:40.637355 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:40.637336 2548 scope.go:117] "RemoveContainer" containerID="9a1dc42e0d8df870ad6d153d1c7e6a3dbc7b1530163cfef0e2b78f1f3d7a6c27" May 25 11:46:43.579845 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:43.579817 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:46:43.580675 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:43.580644 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:46:44.494910 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:44.494874 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 25 11:46:53.580596 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:53.580557 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:46:54.495038 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:46:54.495012 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:47:03.581344 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:03.581306 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:47:05.943308 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943272 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:05.943688 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943673 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" May 25 11:47:05.943731 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943693 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" May 25 11:47:05.943731 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943721 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kube-rbac-proxy" May 25 11:47:05.943792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943731 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kube-rbac-proxy" May 25 11:47:05.943792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943742 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" May 25 11:47:05.943792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943750 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" May 25 11:47:05.943883 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943816 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kube-rbac-proxy" May 25 11:47:05.943883 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943826 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d328ad6c-d586-4706-8021-e4876cd7c03c" containerName="ensemble-graph-20622" May 25 11:47:05.943883 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.943835 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c459914-3782-4a1d-a188-2a60021bb906" containerName="kserve-container" May 25 11:47:05.946689 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.946673 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:05.952556 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.952532 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c3e68-kube-rbac-proxy-sar-config\"" May 25 11:47:05.952678 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.952532 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c3e68-serving-cert\"" May 25 11:47:05.963951 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:05.963931 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:06.092864 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.092835 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.092997 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.092884 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.194208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.194136 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.194208 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.194202 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.194804 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.194781 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.196632 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.196614 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls\") pod \"splitter-graph-c3e68-7bd56c4bbb-xdqlg\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.256794 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.256771 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.384983 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.384955 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:06.387881 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:47:06.387857 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed892d_7b2a_4cf9_bb6d_3933ebe5a850.slice/crio-b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d WatchSource:0}: Error finding container b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d: Status 404 returned error can't find the container with id b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d May 25 11:47:06.650193 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.650157 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" event={"ID":"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850","Type":"ContainerStarted","Data":"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178"} May 25 11:47:06.650193 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.650191 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" event={"ID":"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850","Type":"ContainerStarted","Data":"b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d"} May 25 11:47:06.650366 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.650286 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:06.667975 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:06.667941 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podStartSLOduration=1.6679293290000001 podStartE2EDuration="1.667929329s" podCreationTimestamp="2026-05-25 11:47:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:06.667269967 +0000 UTC m=+1766.678878422" watchObservedRunningTime="2026-05-25 11:47:06.667929329 +0000 UTC m=+1766.679537783" May 25 11:47:12.658859 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:12.658830 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:13.581582 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:13.581538 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" May 25 11:47:16.022754 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.022675 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:16.023156 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.022886 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" containerID="cri-o://24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178" gracePeriod=30 May 25 11:47:16.268835 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.268804 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:47:16.269179 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.269149 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" containerID="cri-o://32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8" gracePeriod=30 May 25 11:47:16.269309 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.269152 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kube-rbac-proxy" containerID="cri-o://27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b" gracePeriod=30 May 25 11:47:16.298516 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.298488 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:47:16.301850 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.301832 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.308358 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.308337 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7f41-predictor-serving-cert\"" May 25 11:47:16.308447 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.308356 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\"" May 25 11:47:16.314243 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.314224 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:47:16.463894 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.463861 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.464047 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.463911 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q84\" (UniqueName: \"kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.464047 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.463967 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.564694 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.564615 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74q84\" (UniqueName: \"kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.564694 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.564674 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.564893 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.564734 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.565421 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.565396 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.567222 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.567203 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.575745 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.575719 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q84\" (UniqueName: \"kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84\") pod \"success-200-isvc-c7f41-predictor-696877f5fd-l8mq2\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.612274 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.612255 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:16.678292 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.678257 2548 generic.go:358] "Generic (PLEG): container finished" podID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerID="27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b" exitCode=2 May 25 11:47:16.678424 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.678338 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerDied","Data":"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b"} May 25 11:47:16.753197 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:16.753166 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:47:16.756191 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:47:16.756165 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68071cf_45b3_407d_a0b0_489687f61b85.slice/crio-2ef0955d37ca95b67ffd31c1e7a6991d20a247d0c4ad60592af42cc721f72529 WatchSource:0}: Error finding container 2ef0955d37ca95b67ffd31c1e7a6991d20a247d0c4ad60592af42cc721f72529: Status 404 returned error can't find the container with id 2ef0955d37ca95b67ffd31c1e7a6991d20a247d0c4ad60592af42cc721f72529 May 25 11:47:17.656943 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.656903 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:17.682302 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.682271 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerStarted","Data":"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251"} May 25 11:47:17.682302 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.682303 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerStarted","Data":"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207"} May 25 11:47:17.682547 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.682314 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerStarted","Data":"2ef0955d37ca95b67ffd31c1e7a6991d20a247d0c4ad60592af42cc721f72529"} May 25 11:47:17.682547 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.682444 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:17.708866 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:17.708777 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podStartSLOduration=1.708765377 podStartE2EDuration="1.708765377s" podCreationTimestamp="2026-05-25 11:47:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:17.708512238 +0000 UTC m=+1777.720120693" watchObservedRunningTime="2026-05-25 11:47:17.708765377 +0000 UTC m=+1777.720373832" May 25 11:47:18.685341 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:18.685267 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:18.686416 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:18.686394 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:47:19.013366 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.013344 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:47:19.184670 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.184639 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgsdc\" (UniqueName: \"kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc\") pod \"ef0629fb-809f-47ab-8312-7f97a370c16b\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " May 25 11:47:19.184816 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.184707 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") pod \"ef0629fb-809f-47ab-8312-7f97a370c16b\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " May 25 11:47:19.184816 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.184736 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"ef0629fb-809f-47ab-8312-7f97a370c16b\" (UID: \"ef0629fb-809f-47ab-8312-7f97a370c16b\") " May 25 11:47:19.185114 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.185073 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-c3e68-kube-rbac-proxy-sar-config") pod "ef0629fb-809f-47ab-8312-7f97a370c16b" (UID: "ef0629fb-809f-47ab-8312-7f97a370c16b"). InnerVolumeSpecName "success-200-isvc-c3e68-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:47:19.186885 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.186863 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc" (OuterVolumeSpecName: "kube-api-access-zgsdc") pod "ef0629fb-809f-47ab-8312-7f97a370c16b" (UID: "ef0629fb-809f-47ab-8312-7f97a370c16b"). InnerVolumeSpecName "kube-api-access-zgsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:47:19.186946 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.186886 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef0629fb-809f-47ab-8312-7f97a370c16b" (UID: "ef0629fb-809f-47ab-8312-7f97a370c16b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:47:19.285419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.285358 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef0629fb-809f-47ab-8312-7f97a370c16b-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:47:19.285419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.285385 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef0629fb-809f-47ab-8312-7f97a370c16b-success-200-isvc-c3e68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:47:19.285419 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.285400 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgsdc\" (UniqueName: \"kubernetes.io/projected/ef0629fb-809f-47ab-8312-7f97a370c16b-kube-api-access-zgsdc\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:47:19.688839 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.688807 2548 generic.go:358] "Generic (PLEG): container finished" podID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerID="32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8" exitCode=0 May 25 11:47:19.689216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.688881 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" May 25 11:47:19.689216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.688903 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerDied","Data":"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8"} May 25 11:47:19.689216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.688943 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt" event={"ID":"ef0629fb-809f-47ab-8312-7f97a370c16b","Type":"ContainerDied","Data":"716e8bc5914ca8935960b1c682d8b5977b47ee2b609d2cdcedc9eb9021c52844"} May 25 11:47:19.689216 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.688957 2548 scope.go:117] "RemoveContainer" containerID="27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b" May 25 11:47:19.689598 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.689573 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:47:19.702036 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.702017 2548 scope.go:117] "RemoveContainer" containerID="32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8" May 25 11:47:19.709472 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.709454 2548 scope.go:117] "RemoveContainer" containerID="27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b" May 25 11:47:19.709713 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:47:19.709696 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b\": container with ID starting with 27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b not found: ID does not exist" containerID="27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b" May 25 11:47:19.709787 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.709724 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b"} err="failed to get container status \"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b\": rpc error: code = NotFound desc = could not find container \"27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b\": container with ID starting with 27f5ff46db5573e8ce64dfd68f08a5cead1c95a2fea21428b3b6ba916841889b not found: ID does not exist" May 25 11:47:19.709787 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.709745 2548 scope.go:117] "RemoveContainer" containerID="32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8" May 25 11:47:19.709977 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:47:19.709960 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8\": container with ID starting with 32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8 not found: ID does not exist" containerID="32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8" May 25 11:47:19.710030 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.709986 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8"} err="failed to get container status \"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8\": rpc error: code = NotFound desc = could not find container \"32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8\": container with ID starting with 32afa6e63a66a504906728b7ae7eba6aec2f9f14e588f7de8c0c48c16736eed8 not found: ID does not exist" May 25 11:47:19.713294 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.713271 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:47:19.718671 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:19.718644 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3e68-predictor-6f8fff67db-sf4vt"] May 25 11:47:20.598282 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:20.598254 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" path="/var/lib/kubelet/pods/ef0629fb-809f-47ab-8312-7f97a370c16b/volumes" May 25 11:47:22.657414 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:22.657377 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:23.581900 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:23.581869 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 11:47:24.693719 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:24.693694 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:47:24.694261 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:24.694234 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:47:27.656460 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:27.656412 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:27.656862 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:27.656533 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:32.656316 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:32.656278 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:34.694589 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:34.694551 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:47:37.656939 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:37.656898 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:40.593484 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:40.593376 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:47:40.616217 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:40.597355 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:47:42.657024 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:42.656984 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:47:44.694474 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:44.694436 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:47:46.059835 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:47:46.059800 2548 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed892d_7b2a_4cf9_bb6d_3933ebe5a850.slice/crio-conmon-24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178.scope\": RecentStats: unable to find data in memory cache]" May 25 11:47:46.060218 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:47:46.059807 2548 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed892d_7b2a_4cf9_bb6d_3933ebe5a850.slice/crio-b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed892d_7b2a_4cf9_bb6d_3933ebe5a850.slice/crio-24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178.scope\": RecentStats: unable to find data in memory cache]" May 25 11:47:46.182043 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.182019 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:46.258976 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.258952 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls\") pod \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " May 25 11:47:46.259092 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.259000 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle\") pod \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\" (UID: \"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850\") " May 25 11:47:46.259415 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.259386 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" (UID: "8bed892d-7b2a-4cf9-bb6d-3933ebe5a850"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:47:46.261128 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.261095 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" (UID: "8bed892d-7b2a-4cf9-bb6d-3933ebe5a850"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:47:46.359731 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.359713 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:47:46.359826 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.359735 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:47:46.763698 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.763630 2548 generic.go:358] "Generic (PLEG): container finished" podID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerID="24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178" exitCode=0 May 25 11:47:46.763698 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.763688 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" May 25 11:47:46.763821 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.763715 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" event={"ID":"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850","Type":"ContainerDied","Data":"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178"} May 25 11:47:46.763821 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.763748 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg" event={"ID":"8bed892d-7b2a-4cf9-bb6d-3933ebe5a850","Type":"ContainerDied","Data":"b171033bea0e97ad35077785a452928f99a42000094d9a42f383cfb66304576d"} May 25 11:47:46.763821 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.763764 2548 scope.go:117] "RemoveContainer" containerID="24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178" May 25 11:47:46.771345 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.771331 2548 scope.go:117] "RemoveContainer" containerID="24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178" May 25 11:47:46.771582 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:47:46.771563 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178\": container with ID starting with 24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178 not found: ID does not exist" containerID="24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178" May 25 11:47:46.771645 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.771594 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178"} err="failed to get container status \"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178\": rpc error: code = NotFound desc = could not find container \"24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178\": container with ID starting with 24de5c1271667be4c04eb5fd7c64dec960f61094be91c3019cccce28a9127178 not found: ID does not exist" May 25 11:47:46.780085 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.780062 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:46.783787 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:46.783768 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c3e68-7bd56c4bbb-xdqlg"] May 25 11:47:48.599365 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:48.599329 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" path="/var/lib/kubelet/pods/8bed892d-7b2a-4cf9-bb6d-3933ebe5a850/volumes" May 25 11:47:54.694492 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:47:54.694458 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" May 25 11:48:04.694726 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:04.694699 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:48:16.256558 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256526 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256802 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kube-rbac-proxy" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256813 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kube-rbac-proxy" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256827 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256833 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256839 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256844 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256913 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kube-rbac-proxy" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256923 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="ef0629fb-809f-47ab-8312-7f97a370c16b" containerName="kserve-container" May 25 11:48:16.256950 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.256933 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="8bed892d-7b2a-4cf9-bb6d-3933ebe5a850" containerName="splitter-graph-c3e68" May 25 11:48:16.259739 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.259723 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.262581 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.262555 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c7f41-kube-rbac-proxy-sar-config\"" May 25 11:48:16.262747 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.262730 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c7f41-serving-cert\"" May 25 11:48:16.275759 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.275740 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:48:16.363253 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.363224 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.363369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.363262 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.464008 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.463979 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.464140 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.464028 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.464245 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:48:16.464232 2548 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-c7f41-serving-cert: secret "splitter-graph-c7f41-serving-cert" not found May 25 11:48:16.464298 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:48:16.464289 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls podName:41ab2c3d-0282-44d9-bbdd-c2f34184c28c nodeName:}" failed. No retries permitted until 2026-05-25 11:48:16.964273245 +0000 UTC m=+1836.975881682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls") pod "splitter-graph-c7f41-7c9cdb5df5-28c54" (UID: "41ab2c3d-0282-44d9-bbdd-c2f34184c28c") : secret "splitter-graph-c7f41-serving-cert" not found May 25 11:48:16.464588 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.464568 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.968717 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.968673 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:16.971235 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:16.971202 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") pod \"splitter-graph-c7f41-7c9cdb5df5-28c54\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:17.169349 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:17.169316 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:17.297072 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:17.297048 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:48:17.299599 ip-10-0-141-33 kubenswrapper[2548]: W0525 11:48:17.299568 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ab2c3d_0282_44d9_bbdd_c2f34184c28c.slice/crio-4f50869c47b0b41fdd6a88787d1387089ccdb5dcec4e70372954844756226481 WatchSource:0}: Error finding container 4f50869c47b0b41fdd6a88787d1387089ccdb5dcec4e70372954844756226481: Status 404 returned error can't find the container with id 4f50869c47b0b41fdd6a88787d1387089ccdb5dcec4e70372954844756226481 May 25 11:48:17.843792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:17.843753 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" event={"ID":"41ab2c3d-0282-44d9-bbdd-c2f34184c28c","Type":"ContainerStarted","Data":"8e1ccf8f266dc3824006bff69585231596e6b5e53e38740ec26ba84d179bf72b"} May 25 11:48:17.843792 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:17.843790 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" event={"ID":"41ab2c3d-0282-44d9-bbdd-c2f34184c28c","Type":"ContainerStarted","Data":"4f50869c47b0b41fdd6a88787d1387089ccdb5dcec4e70372954844756226481"} May 25 11:48:17.844042 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:17.843815 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:23.852822 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:23.852788 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:48:23.868768 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:48:23.868726 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podStartSLOduration=7.868713215 podStartE2EDuration="7.868713215s" podCreationTimestamp="2026-05-25 11:48:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:48:17.86908053 +0000 UTC m=+1837.880688984" watchObservedRunningTime="2026-05-25 11:48:23.868713215 +0000 UTC m=+1843.880321670" May 25 11:52:40.612300 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:52:40.612270 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:52:40.616039 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:52:40.616016 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:56:30.912217 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:30.912129 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:56:30.912742 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:30.912405 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" containerID="cri-o://8e1ccf8f266dc3824006bff69585231596e6b5e53e38740ec26ba84d179bf72b" gracePeriod=30 May 25 11:56:31.003255 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:31.003218 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:56:31.003555 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:31.003522 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" containerID="cri-o://5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207" gracePeriod=30 May 25 11:56:31.003698 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:31.003571 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kube-rbac-proxy" containerID="cri-o://52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251" gracePeriod=30 May 25 11:56:31.199903 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:31.199809 2548 generic.go:358] "Generic (PLEG): container finished" podID="d68071cf-45b3-407d-a0b0-489687f61b85" containerID="52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251" exitCode=2 May 25 11:56:31.199903 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:31.199888 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerDied","Data":"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251"} May 25 11:56:33.851576 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:33.851527 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:56:34.348070 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.348046 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:56:34.529934 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.529837 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls\") pod \"d68071cf-45b3-407d-a0b0-489687f61b85\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " May 25 11:56:34.530094 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.529957 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74q84\" (UniqueName: \"kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84\") pod \"d68071cf-45b3-407d-a0b0-489687f61b85\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " May 25 11:56:34.530094 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.529993 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"d68071cf-45b3-407d-a0b0-489687f61b85\" (UID: \"d68071cf-45b3-407d-a0b0-489687f61b85\") " May 25 11:56:34.530418 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.530392 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-c7f41-kube-rbac-proxy-sar-config") pod "d68071cf-45b3-407d-a0b0-489687f61b85" (UID: "d68071cf-45b3-407d-a0b0-489687f61b85"). InnerVolumeSpecName "success-200-isvc-c7f41-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:56:34.532236 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.532211 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84" (OuterVolumeSpecName: "kube-api-access-74q84") pod "d68071cf-45b3-407d-a0b0-489687f61b85" (UID: "d68071cf-45b3-407d-a0b0-489687f61b85"). InnerVolumeSpecName "kube-api-access-74q84". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 11:56:34.532340 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.532240 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d68071cf-45b3-407d-a0b0-489687f61b85" (UID: "d68071cf-45b3-407d-a0b0-489687f61b85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:56:34.630732 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.630696 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q84\" (UniqueName: \"kubernetes.io/projected/d68071cf-45b3-407d-a0b0-489687f61b85-kube-api-access-74q84\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:56:34.630732 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.630726 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68071cf-45b3-407d-a0b0-489687f61b85-success-200-isvc-c7f41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:56:34.630732 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:34.630737 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68071cf-45b3-407d-a0b0-489687f61b85-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:56:35.211912 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.211873 2548 generic.go:358] "Generic (PLEG): container finished" podID="d68071cf-45b3-407d-a0b0-489687f61b85" containerID="5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207" exitCode=0 May 25 11:56:35.212369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.211953 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" May 25 11:56:35.212369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.211991 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerDied","Data":"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207"} May 25 11:56:35.212369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.212027 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2" event={"ID":"d68071cf-45b3-407d-a0b0-489687f61b85","Type":"ContainerDied","Data":"2ef0955d37ca95b67ffd31c1e7a6991d20a247d0c4ad60592af42cc721f72529"} May 25 11:56:35.212369 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.212048 2548 scope.go:117] "RemoveContainer" containerID="52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251" May 25 11:56:35.220109 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.220071 2548 scope.go:117] "RemoveContainer" containerID="5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207" May 25 11:56:35.227803 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.227769 2548 scope.go:117] "RemoveContainer" containerID="52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251" May 25 11:56:35.228087 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:56:35.228060 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251\": container with ID starting with 52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251 not found: ID does not exist" containerID="52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251" May 25 11:56:35.228206 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.228112 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251"} err="failed to get container status \"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251\": rpc error: code = NotFound desc = could not find container \"52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251\": container with ID starting with 52d6e146184367fb6f95a9fd299654ea397fb08e7f99cbe5f4fa6ee1f0dec251 not found: ID does not exist" May 25 11:56:35.228206 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.228140 2548 scope.go:117] "RemoveContainer" containerID="5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207" May 25 11:56:35.228310 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.228291 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:56:35.228420 ip-10-0-141-33 kubenswrapper[2548]: E0525 11:56:35.228402 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207\": container with ID starting with 5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207 not found: ID does not exist" containerID="5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207" May 25 11:56:35.228471 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.228430 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207"} err="failed to get container status \"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207\": rpc error: code = NotFound desc = could not find container \"5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207\": container with ID starting with 5322f78de3e0b96f19861d835fe7fea515cc01abe416ee6a5103a41c7bd0a207 not found: ID does not exist" May 25 11:56:35.230966 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:35.230941 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7f41-predictor-696877f5fd-l8mq2"] May 25 11:56:36.598550 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:36.598513 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" path="/var/lib/kubelet/pods/d68071cf-45b3-407d-a0b0-489687f61b85/volumes" May 25 11:56:38.851628 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:38.851588 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:56:43.851924 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:43.851886 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:56:43.852328 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:43.851989 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:56:48.851738 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:48.851692 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:56:53.851964 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:53.851920 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:56:58.851360 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:56:58.851318 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 11:57:01.283116 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.283008 2548 generic.go:358] "Generic (PLEG): container finished" podID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerID="8e1ccf8f266dc3824006bff69585231596e6b5e53e38740ec26ba84d179bf72b" exitCode=0 May 25 11:57:01.283116 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.283058 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" event={"ID":"41ab2c3d-0282-44d9-bbdd-c2f34184c28c","Type":"ContainerDied","Data":"8e1ccf8f266dc3824006bff69585231596e6b5e53e38740ec26ba84d179bf72b"} May 25 11:57:01.557772 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.557747 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:57:01.615890 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.615836 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") pod \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " May 25 11:57:01.616086 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.615945 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle\") pod \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\" (UID: \"41ab2c3d-0282-44d9-bbdd-c2f34184c28c\") " May 25 11:57:01.616326 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.616304 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "41ab2c3d-0282-44d9-bbdd-c2f34184c28c" (UID: "41ab2c3d-0282-44d9-bbdd-c2f34184c28c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 11:57:01.618093 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.618071 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41ab2c3d-0282-44d9-bbdd-c2f34184c28c" (UID: "41ab2c3d-0282-44d9-bbdd-c2f34184c28c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 11:57:01.716584 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.716544 2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-openshift-service-ca-bundle\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:57:01.716584 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:01.716578 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ab2c3d-0282-44d9-bbdd-c2f34184c28c-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 11:57:02.287156 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.287092 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" event={"ID":"41ab2c3d-0282-44d9-bbdd-c2f34184c28c","Type":"ContainerDied","Data":"4f50869c47b0b41fdd6a88787d1387089ccdb5dcec4e70372954844756226481"} May 25 11:57:02.287156 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.287155 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54" May 25 11:57:02.287796 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.287168 2548 scope.go:117] "RemoveContainer" containerID="8e1ccf8f266dc3824006bff69585231596e6b5e53e38740ec26ba84d179bf72b" May 25 11:57:02.307537 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.307509 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:57:02.309818 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.309796 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c7f41-7c9cdb5df5-28c54"] May 25 11:57:02.599578 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:02.599494 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" path="/var/lib/kubelet/pods/41ab2c3d-0282-44d9-bbdd-c2f34184c28c/volumes" May 25 11:57:40.629720 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:40.629609 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 11:57:40.633809 ip-10-0-141-33 kubenswrapper[2548]: I0525 11:57:40.633704 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 12:02:40.645791 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:02:40.645676 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 12:02:40.651067 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:02:40.651047 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 12:03:54.474915 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:54.474835 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 12:03:54.475421 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:54.475138 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" containerID="cri-o://946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1" gracePeriod=30 May 25 12:03:54.475421 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:54.475189 2548 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kube-rbac-proxy" containerID="cri-o://d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca" gracePeriod=30 May 25 12:03:55.401176 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:55.401141 2548 generic.go:358] "Generic (PLEG): container finished" podID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerID="d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca" exitCode=2 May 25 12:03:55.401340 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:55.401204 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerDied","Data":"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca"} May 25 12:03:56.910118 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.910075 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 12:03:56.971697 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.971672 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " May 25 12:03:56.971807 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.971765 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8svk\" (UniqueName: \"kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk\") pod \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " May 25 12:03:56.971807 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.971799 2548 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls\") pod \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\" (UID: \"38d5c0a0-73a1-4d64-84ec-54ef72df2661\") " May 25 12:03:56.972029 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.972009 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-c1cab-kube-rbac-proxy-sar-config") pod "38d5c0a0-73a1-4d64-84ec-54ef72df2661" (UID: "38d5c0a0-73a1-4d64-84ec-54ef72df2661"). InnerVolumeSpecName "success-200-isvc-c1cab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 12:03:56.973925 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.973892 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk" (OuterVolumeSpecName: "kube-api-access-l8svk") pod "38d5c0a0-73a1-4d64-84ec-54ef72df2661" (UID: "38d5c0a0-73a1-4d64-84ec-54ef72df2661"). InnerVolumeSpecName "kube-api-access-l8svk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 12:03:56.974004 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:56.973901 2548 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38d5c0a0-73a1-4d64-84ec-54ef72df2661" (UID: "38d5c0a0-73a1-4d64-84ec-54ef72df2661"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 12:03:57.072723 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.072673 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8svk\" (UniqueName: \"kubernetes.io/projected/38d5c0a0-73a1-4d64-84ec-54ef72df2661-kube-api-access-l8svk\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 12:03:57.072723 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.072692 2548 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38d5c0a0-73a1-4d64-84ec-54ef72df2661-proxy-tls\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 12:03:57.072723 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.072703 2548 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38d5c0a0-73a1-4d64-84ec-54ef72df2661-success-200-isvc-c1cab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-33.ec2.internal\" DevicePath \"\"" May 25 12:03:57.407421 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.407389 2548 generic.go:358] "Generic (PLEG): container finished" podID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerID="946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1" exitCode=0 May 25 12:03:57.407546 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.407483 2548 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" May 25 12:03:57.407546 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.407527 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerDied","Data":"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1"} May 25 12:03:57.407621 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.407577 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r" event={"ID":"38d5c0a0-73a1-4d64-84ec-54ef72df2661","Type":"ContainerDied","Data":"0b669b70479712fd84ea357abca2d48388679557ed7f52a700f8c784d65b1003"} May 25 12:03:57.407621 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.407597 2548 scope.go:117] "RemoveContainer" containerID="d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca" May 25 12:03:57.417069 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.417046 2548 scope.go:117] "RemoveContainer" containerID="946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1" May 25 12:03:57.424568 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.424550 2548 scope.go:117] "RemoveContainer" containerID="d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca" May 25 12:03:57.424800 ip-10-0-141-33 kubenswrapper[2548]: E0525 12:03:57.424782 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca\": container with ID starting with d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca not found: ID does not exist" containerID="d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca" May 25 12:03:57.424862 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.424807 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca"} err="failed to get container status \"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca\": rpc error: code = NotFound desc = could not find container \"d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca\": container with ID starting with d773e59fa15386a34b98246b44f154a484b2442236124db391e206356a8109ca not found: ID does not exist" May 25 12:03:57.424862 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.424825 2548 scope.go:117] "RemoveContainer" containerID="946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1" May 25 12:03:57.425085 ip-10-0-141-33 kubenswrapper[2548]: E0525 12:03:57.425060 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1\": container with ID starting with 946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1 not found: ID does not exist" containerID="946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1" May 25 12:03:57.425152 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.425093 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1"} err="failed to get container status \"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1\": rpc error: code = NotFound desc = could not find container \"946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1\": container with ID starting with 946e92195bc2adfb6c3d706ee82958ac42f790ef984a9f68b0f5d1424c4861b1 not found: ID does not exist" May 25 12:03:57.429798 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.429774 2548 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 12:03:57.433273 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:57.433247 2548 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1cab-predictor-7774d7cc9b-qvk9r"] May 25 12:03:58.599242 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:03:58.599204 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" path="/var/lib/kubelet/pods/38d5c0a0-73a1-4d64-84ec-54ef72df2661/volumes" May 25 12:04:22.812704 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.812662 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzvv7/must-gather-ph9tc"] May 25 12:04:22.813412 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813375 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813426 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813460 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813470 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813495 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813505 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813516 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kube-rbac-proxy" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813524 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kube-rbac-proxy" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813539 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kube-rbac-proxy" May 25 12:04:22.813561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813547 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kube-rbac-proxy" May 25 12:04:22.813989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813751 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="41ab2c3d-0282-44d9-bbdd-c2f34184c28c" containerName="splitter-graph-c7f41" May 25 12:04:22.813989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813762 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kserve-container" May 25 12:04:22.813989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813781 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kube-rbac-proxy" May 25 12:04:22.813989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813792 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="38d5c0a0-73a1-4d64-84ec-54ef72df2661" containerName="kube-rbac-proxy" May 25 12:04:22.813989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.813805 2548 memory_manager.go:356] "RemoveStaleState removing state" podUID="d68071cf-45b3-407d-a0b0-489687f61b85" containerName="kserve-container" May 25 12:04:22.817434 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.817410 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:22.820415 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.820389 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzvv7\"/\"kube-root-ca.crt\"" May 25 12:04:22.820756 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.820722 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/must-gather-ph9tc"] May 25 12:04:22.821485 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.821452 2548 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fzvv7\"/\"default-dockercfg-mcf27\"" May 25 12:04:22.821610 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.821460 2548 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzvv7\"/\"openshift-service-ca.crt\"" May 25 12:04:22.947555 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.947526 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fbn\" (UniqueName: \"kubernetes.io/projected/07cdc081-2265-43f9-9f13-e981005a8a1e-kube-api-access-b4fbn\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:22.947555 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:22.947558 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07cdc081-2265-43f9-9f13-e981005a8a1e-must-gather-output\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.048613 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.048570 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fbn\" (UniqueName: \"kubernetes.io/projected/07cdc081-2265-43f9-9f13-e981005a8a1e-kube-api-access-b4fbn\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.048613 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.048619 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07cdc081-2265-43f9-9f13-e981005a8a1e-must-gather-output\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.048983 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.048958 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07cdc081-2265-43f9-9f13-e981005a8a1e-must-gather-output\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.056894 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.056868 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fbn\" (UniqueName: \"kubernetes.io/projected/07cdc081-2265-43f9-9f13-e981005a8a1e-kube-api-access-b4fbn\") pod \"must-gather-ph9tc\" (UID: \"07cdc081-2265-43f9-9f13-e981005a8a1e\") " pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.127915 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.127890 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" May 25 12:04:23.258319 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.258285 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/must-gather-ph9tc"] May 25 12:04:23.261335 ip-10-0-141-33 kubenswrapper[2548]: W0525 12:04:23.261309 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07cdc081_2265_43f9_9f13_e981005a8a1e.slice/crio-da84b66751fa95d67eec67374f9822910b3196353295bfd93197c1809cc70450 WatchSource:0}: Error finding container da84b66751fa95d67eec67374f9822910b3196353295bfd93197c1809cc70450: Status 404 returned error can't find the container with id da84b66751fa95d67eec67374f9822910b3196353295bfd93197c1809cc70450 May 25 12:04:23.262983 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.262966 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 12:04:23.480465 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:23.480398 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" event={"ID":"07cdc081-2265-43f9-9f13-e981005a8a1e","Type":"ContainerStarted","Data":"da84b66751fa95d67eec67374f9822910b3196353295bfd93197c1809cc70450"} May 25 12:04:24.484764 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:24.484677 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" event={"ID":"07cdc081-2265-43f9-9f13-e981005a8a1e","Type":"ContainerStarted","Data":"7af53bb9ca8c16ff3382994d09fe3b4c5659e1f1d853d1fa0756098e75728141"} May 25 12:04:24.484764 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:24.484722 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" event={"ID":"07cdc081-2265-43f9-9f13-e981005a8a1e","Type":"ContainerStarted","Data":"1e650b383af7eb2d63ee6997599c9385851377a9129b410434ac3591257dda74"} May 25 12:04:24.500896 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:24.500605 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fzvv7/must-gather-ph9tc" podStartSLOduration=1.632235549 podStartE2EDuration="2.500586498s" podCreationTimestamp="2026-05-25 12:04:22 +0000 UTC" firstStartedPulling="2026-05-25 12:04:23.263141527 +0000 UTC m=+2803.274749962" lastFinishedPulling="2026-05-25 12:04:24.131492471 +0000 UTC m=+2804.143100911" observedRunningTime="2026-05-25 12:04:24.499284032 +0000 UTC m=+2804.510892488" watchObservedRunningTime="2026-05-25 12:04:24.500586498 +0000 UTC m=+2804.512194953" May 25 12:04:25.525895 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:25.525848 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8wx4h_79fe35ea-6771-4a9e-ab6f-3790e1c579a1/global-pull-secret-syncer/0.log" May 25 12:04:25.737764 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:25.737733 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mjfpc_ad9ab7bf-e6d3-426c-b726-6c7e90cb1435/konnectivity-agent/0.log" May 25 12:04:25.805724 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:25.805654 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-33.ec2.internal_b41372adabb94dc7bf44dd30836c4051/haproxy/0.log" May 25 12:04:29.586228 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:29.586198 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99c7k_e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92/node-exporter/0.log" May 25 12:04:29.613429 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:29.613389 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99c7k_e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92/kube-rbac-proxy/0.log" May 25 12:04:29.634800 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:29.634770 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99c7k_e1c8aa98-ec36-47fa-8d8f-f14b8c5a6a92/init-textfile/0.log" May 25 12:04:31.422689 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:31.422660 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-vx9c7_23080302-2bff-48d8-a38a-df55f540ae20/networking-console-plugin/0.log" May 25 12:04:32.507889 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.507856 2548 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx"] May 25 12:04:32.513090 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.513062 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.521678 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.521382 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx"] May 25 12:04:32.627753 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.627693 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-lib-modules\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.627946 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.627774 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-sys\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.627946 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.627807 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpt9d\" (UniqueName: \"kubernetes.io/projected/6afa48e8-a778-4488-923e-72db4226e59d-kube-api-access-vpt9d\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.627946 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.627837 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-proc\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.627946 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.627900 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-podres\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728714 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728673 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vpt9d\" (UniqueName: \"kubernetes.io/projected/6afa48e8-a778-4488-923e-72db4226e59d-kube-api-access-vpt9d\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728714 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728713 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-proc\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728937 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728761 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-podres\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728937 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728785 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-lib-modules\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728937 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728813 2548 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-sys\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728937 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728871 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-proc\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.728937 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728879 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-sys\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.729197 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728935 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-podres\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.729197 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.728945 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6afa48e8-a778-4488-923e-72db4226e59d-lib-modules\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.737680 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.737644 2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpt9d\" (UniqueName: \"kubernetes.io/projected/6afa48e8-a778-4488-923e-72db4226e59d-kube-api-access-vpt9d\") pod \"perf-node-gather-daemonset-kwrjx\" (UID: \"6afa48e8-a778-4488-923e-72db4226e59d\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.825725 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.825646 2548 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:32.960066 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:32.960014 2548 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx"] May 25 12:04:32.964800 ip-10-0-141-33 kubenswrapper[2548]: W0525 12:04:32.964769 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6afa48e8_a778_4488_923e_72db4226e59d.slice/crio-26f67d633fa086413b12c1ca3afc457f06fa5b17cf66d97c0a28ae6c071e5404 WatchSource:0}: Error finding container 26f67d633fa086413b12c1ca3afc457f06fa5b17cf66d97c0a28ae6c071e5404: Status 404 returned error can't find the container with id 26f67d633fa086413b12c1ca3afc457f06fa5b17cf66d97c0a28ae6c071e5404 May 25 12:04:33.448773 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.448740 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f9bhg_8bb17520-93f2-4787-a784-45ecb9dca246/dns/0.log" May 25 12:04:33.469480 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.469441 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f9bhg_8bb17520-93f2-4787-a784-45ecb9dca246/kube-rbac-proxy/0.log" May 25 12:04:33.515192 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.515151 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" event={"ID":"6afa48e8-a778-4488-923e-72db4226e59d","Type":"ContainerStarted","Data":"70d9c60be34c36635fd8c1ee212c15d331920396c3fe9cb8285450a50a4e0c00"} May 25 12:04:33.515556 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.515198 2548 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" event={"ID":"6afa48e8-a778-4488-923e-72db4226e59d","Type":"ContainerStarted","Data":"26f67d633fa086413b12c1ca3afc457f06fa5b17cf66d97c0a28ae6c071e5404"} May 25 12:04:33.515556 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.515309 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:33.531821 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.531779 2548 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" podStartSLOduration=1.531767673 podStartE2EDuration="1.531767673s" podCreationTimestamp="2026-05-25 12:04:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:04:33.531270031 +0000 UTC m=+2813.542878493" watchObservedRunningTime="2026-05-25 12:04:33.531767673 +0000 UTC m=+2813.543376127" May 25 12:04:33.562249 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:33.562222 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jf4zc_06d68c20-0bdc-4cc2-8357-885bba30014f/dns-node-resolver/0.log" May 25 12:04:34.023177 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:34.023091 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bc79c8b59-mfwq2_9baa0b4d-a60a-468d-a4dc-d8f52596706e/registry/0.log" May 25 12:04:34.099878 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:34.099850 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwh8j_2f9b0c54-3b7b-44e1-9614-0890e21e2157/node-ca/0.log" May 25 12:04:35.265989 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:35.265950 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kkhvk_fac0286b-34ac-45e8-b336-98b16189b98e/serve-healthcheck-canary/0.log" May 25 12:04:35.647390 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:35.647359 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zwgb_859924b5-0863-4e63-969c-fb4bb29d3863/kube-rbac-proxy/0.log" May 25 12:04:35.669123 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:35.669084 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zwgb_859924b5-0863-4e63-969c-fb4bb29d3863/exporter/0.log" May 25 12:04:35.698237 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:35.698206 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zwgb_859924b5-0863-4e63-969c-fb4bb29d3863/extractor/0.log" May 25 12:04:38.351787 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:38.351750 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-qwdxm_c016ee30-d2ed-4386-a9c9-82b758c78e17/manager/0.log" May 25 12:04:38.402879 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:38.402837 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-4tbr2_3b9d3ad8-248b-4eb7-b622-f7d30fe35b44/seaweedfs/0.log" May 25 12:04:39.529352 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:39.529325 2548 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-kwrjx" May 25 12:04:43.740733 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.740705 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/kube-multus-additional-cni-plugins/0.log" May 25 12:04:43.766174 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.766142 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/egress-router-binary-copy/0.log" May 25 12:04:43.786287 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.786257 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/cni-plugins/0.log" May 25 12:04:43.807925 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.807897 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/bond-cni-plugin/0.log" May 25 12:04:43.831427 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.831388 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/routeoverride-cni/0.log" May 25 12:04:43.851454 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.851432 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/whereabouts-cni-bincopy/0.log" May 25 12:04:43.874066 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:43.874040 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-58hq2_82daf91e-0f98-4b15-ba16-d1732291b935/whereabouts-cni/0.log" May 25 12:04:44.319852 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:44.319775 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rqdbv_404ce4d8-0b4a-408e-9e3c-a45b682341b0/kube-multus/0.log" May 25 12:04:44.381593 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:44.381562 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-987s2_8a17bca6-35d7-416a-ab02-2a1c52de2b5a/network-metrics-daemon/0.log" May 25 12:04:44.400211 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:44.400190 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-987s2_8a17bca6-35d7-416a-ab02-2a1c52de2b5a/kube-rbac-proxy/0.log" May 25 12:04:45.806033 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.805987 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-controller/0.log" May 25 12:04:45.824902 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.824865 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/0.log" May 25 12:04:45.855667 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.855630 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovn-acl-logging/1.log" May 25 12:04:45.876561 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.876530 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/kube-rbac-proxy-node/0.log" May 25 12:04:45.901203 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.901172 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/kube-rbac-proxy-ovn-metrics/0.log" May 25 12:04:45.924820 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.924789 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/northd/0.log" May 25 12:04:45.950214 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.950174 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/nbdb/0.log" May 25 12:04:45.972813 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:45.972783 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/sbdb/0.log" May 25 12:04:46.178120 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:46.178076 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tmtvc_1b22b242-6a7a-484a-ad28-64199fd493e0/ovnkube-controller/0.log" May 25 12:04:47.175226 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:47.175191 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-czj9c_8d60e59c-fecb-4f0f-aa3d-f29793dbbfc1/network-check-target-container/0.log" May 25 12:04:48.229365 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:48.229330 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-n6pld_c92149ff-8de4-436e-95fe-05c9a447df6d/iptables-alerter/0.log" May 25 12:04:48.970144 ip-10-0-141-33 kubenswrapper[2548]: I0525 12:04:48.970026 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rb5pn_7a444b8d-a75f-49cb-916d-0be57c841be6/tuned/0.log"